The Traps: Noul vs Choice, Score Expectations, and Thresholds That Don't Transfer

The assumption that breaks systems

You ship a Noul asking whether a customer wants a refund. It works. Later you need three outcomes instead of two, so you convert it to a Choice with refund, exchange, and information. The instruction text is nearly identical. You keep your threshold of 0.5, now applied to probabilities["refund"], and ship it.

You have just introduced a bug that no test will catch, because both versions return plausible numbers and neither errors.

TypeSafe documents this under common-sense structural invariants, failure mode 8 on the jaggedness page for jev-1.13. It is the least intuitive entry on that list and the most expensive to learn in production, so it is worth walking through the published measurements carefully.

The headline from the docs, and it is a deliberately two-sided statement: jev-1.13 is extremely consistent, meaning you should expect quantitatively similar outputs for semantically similar inputs. But many structural invariants one might imagine to hold simply are not guaranteed by the model.

Consistency and algebra are different properties. Jev has the first. It does not promise the second.

Measurement one: the same question, two types

TypeSafe asked "Is the customer asking for a refund?" about this ticket:

"I'm not happy with the fit. What are my options here?"

Once as a Noul. Once as a yes/no Choice. Recorded results:

Sit with the gap. The Noul says 0.22, which is a soft no with real residual probability on yes. If your threshold were 0.2, the Noul fires. The Choice puts 0.01 on yes, a hard no, with 0.97 confidence. Same model, same ticket, same underlying question, and the two answers land in different decision regions for any threshold between roughly 0.02 and 0.22.

The docs are careful about which numbers are even comparable: "The comparable numbers are noul and probabilities["yes"], and it is not obvious how to interpret either the Choice output and confidence for the Noul question or vice versa."

Note especially what confidence does not mean here. A Choice's confidence of 0.97 describes how peaked the distribution is across yes and no. It is not the probability of the answer, and it has no counterpart on a Noul at all. Noul answers carry no confidence field, because a two-outcome distribution is fully described by the single noul value. Comparing a Noul of 0.22 to a confidence of 0.97 is comparing two different kinds of quantity.

Measurement two: a question and its negation

The second published measurement is stranger and more instructive. Two Nouls, one the negation of the other, on this ticket:

"I was charged twice for the same order. Can someone look into this?"

The first asks whether the customer is asking for a refund. The second asks whether the customer is asking for something other than a refund.

The sum is 1.19. If you expected P(A) + P(not A) = 1, this is a 19-point violation.

The docs note there are many reasons P(noul) and 1 - P(not noul) may not be directly comparable. The mechanical reason is that these are two independent evaluations of two different English sentences, not two views of one internal probability. Every question in a request is evaluated independently; one answer is never hidden context for another. "Asking for a refund" and "asking for something other than a refund" are not logical complements in natural language either. This ticket is a good example: reporting a duplicate charge and asking someone to look into it plausibly satisfies both readings at once, which is exactly what a 0.72 and a 0.47 look like.

The practical consequence: you cannot derive one answer from another. If you need both propositions, ask both and handle the overlap explicitly in code. If you need them to be mutually exclusive, that is what a Choice is for.

Why Choice and Noul answer different questions

The sentence from the docs that explains both measurements at once is worth keeping:

"the Choice is relative, settling which option, while each Noul is absolute and can be low for all of them."

A Choice distributes probability across your options. The probabilities sum to 1 by construction, because the model is deciding which of these, given that it must pick one. Supply three bad options and one of them still gets a high probability. The Choice cannot tell you that none of them fit unless you gave it an other option to say so.

A Noul evaluates one proposition on its own terms. Nothing forces a set of Nouls to sum to anything. Ask one Noul per option and they can all come back at 0.1, which is the model telling you none of these apply, a statement a Choice structurally cannot make.

This is why the docs describe using both on the same shortlist in the skill suggestion cookbook: the Choice to pick a skill, and Nouls to decide whether to suggest one at all. Those are genuinely different questions. Which one is best and is any of them good enough need different primitives, and running both is the correct design rather than a redundant one.

# Which category fits best? (relative, sums to 1)
"category": Choice(
    instructions="Which category best describes this ticket?",
    criteria={"billing": "...", "technical": "...", "account": "..."},
),
# Is it actually any of them? (absolute, independent)
"fits_billing":   Noul(instructions="Is this ticket about a charge, invoice, or subscription?"),
"fits_technical": Noul(instructions="Is this ticket about a bug, outage, or integration failure?"),

best = response.answers["category"]
fits = response.answers[f"fits_{best.choice}"].noul
 
if fits < 0.4:
    # The Choice picked a winner, but nothing actually fits. Escalate.
    route_to_human(ticket_id)

That guard is impossible with a Choice alone. The Choice will always name a winner.

Score expectations are not measurements

The second trap, from the math and numbers section of the same page, catches people who are being careful rather than careless.

A Score returns an expectation over your levels. With three levels the value runs 0 to 2, and it can land anywhere in between. The arithmetic is plain: the score is each level number multiplied by its probability, summed. In the documented bug severity example, probabilities of 0.0, 0.57, and 0.43 across levels 0, 1, and 2 give 0 × 0.0 + 1 × 0.57 + 2 × 0.43 = 1.43.

So a score of 1.43 is not a judgment that the bug sits 43% of the way between "workaround exists" and "no workaround". It is a summary of a split: the model put 0.57 on level 1 and 0.43 on level 2. Confidence on that answer was 0.35, precisely because it was split.

This matters because an expectation between levels has two completely different causes that the single number cannot distinguish. The state might genuinely sit between your levels. Or the model might be torn between two levels for a state that is squarely one or the other. A score of 1.0 with all probability on level 1 and a score of 1.0 from a 50/50 split between levels 0 and 2 are the same number and mean opposite things. That is what probabilities and confidence are for. Read them.

TypeSafe's guidance is explicit, and the reason it gives is the load-bearing part: do not use score outputs to compute the exact magnitude of a number between two levels of a criterion. You can use the expectation to check whether it passes a threshold. But, in the docs' words, "jev-1.13's score levels are weak in numerical calibration", and it will not let you reconstruct an exact number by interpolating between the nearest two levels.

The tempting antipattern:

# WRONG. The levels are not a calibrated number line.
levels_usd = [0, 50_000, 250_000, 1_000_000]
score = response.answers["deal_size"].score          # e.g. 1.6
lo, hi = int(score), min(int(score) + 1, len(levels_usd) - 1)
frac = score - lo
estimated_usd = levels_usd[lo] + frac * (levels_usd[hi] - levels_usd[lo])
# -> $170,000, a number with no basis in anything the model measured

That output looks like data. It will end up in a dashboard. Nothing in the model licenses it.

What the expectation is good for is ordering and thresholding:

severity = response.answers["bug_severity"]
 
# Fine: a threshold on the expectation.
if severity.score > 1.5:
    escalate(ticket_id)
 
# Better: consult the distribution when the decision is expensive.
if severity.probabilities["2"] > 0.3:
    # Meaningful mass on "blocking, no workaround" even if the expectation is mid-scale.
    escalate(ticket_id)
 
# Also fine: composite scoring normalizes and weights expectations.
normalized = severity.score / 2

If you need an actual dollar figure, a date, or a count, extract it with a Choice over enumerated options or a regex and compute in code. That is the general shape of the fix for everything in the math and numbers section.

Thresholds do not transfer

Three corollaries follow from all of the above, and each one is a real migration hazard.

Across question types. A 0.5 cutoff tuned on a Noul means nothing applied to probabilities["yes"] of a Choice. The first measurement above shows those quantities landing at 0.22 and 0.01 on the same input. The docs say it directly: don't carry a threshold tuned on a Noul over to a Choice.

Across changes to a Choice's option set. Choice probabilities are relative to the options you supplied. Add a fourth option and every probability is renormalized against a different denominator. Your tuned cutoff on billing was calibrated against three competitors; now there are four. Adding an option is a threshold-invalidating change even when the text of the other options is untouched.

Across model versions. This is the one with a concrete mitigation. jev-latest resolves to jev-1.13.0 today, and an alias moves when a new release ships, so answers behind it can change with no change on your side. The docs give the rule plainly: if you have tuned confidence thresholds against a specific version, pin that version's ID instead of the alias and move to the new one on your own schedule.

# Pin, so your thresholds keep meaning what you measured them to mean.
client = TypeSafeClient(model="jev-1.13.0")

The response's model field reports the versioned ID that answered. Log it alongside every stored decision. When you do eventually migrate, that log is what lets you re-measure rather than re-guess.

A rule that covers all of it

Ask each decision one way, and enforce identities in code.

The first half means picking one primitive per decision and staying on it. If a decision is yes/no, it is a Noul and it stays a Noul; converting it to a Choice is a re-tuning event, not a refactor. If it is one-of-N, it is a Choice from the start, with an other option so the model can decline.

The second half means any relationship you need to hold gets written in Python, not assumed of the model. If two outcomes must be mutually exclusive, make them options in one Choice, or compute the complement yourself from a single Noul:

# Don't ask for the negation. Derive it.
p_refund = response.answers["refund_requested"].noul
p_not_refund = 1.0 - p_refund     # exact, by construction

Any invariant you want, you build. The model gives you calibrated judgment over text. It does not give you algebra, and the published numbers are TypeSafe telling you so in advance.

What to do on Monday

Three concrete moves, in order of payoff.

Grep your codebase for thresholds applied to answers whose question type changed at some point in the repo's history. Those are live bugs sitting behind plausible numbers.

Find every place you interpolate a Score expectation into a real-world quantity and delete it. Replace with a Choice over enumerated buckets if you need the value, or a threshold if you only need a decision.

Pin jev-1.13.0 everywhere you have tuned a number against measured behavior, and log the response's model field with every stored decision. Then, when 1.14 ships, you will be able to tell what changed instead of discovering it from a support queue.