An agent harness spends far more of its time deciding than writing. Before the model writes a single word, the harness has to decide which skill should handle the request, whether the request is something the agent should touch at all, and whether a tool call is safe to run. After the model answers, it has to decide whether the task is finished and whether a person should look at the result before it goes out.

Many teams send each of these small decisions to a large language model and parse whatever text comes back. That works in a demo. In production it raises two problems. The answer might not be one of the values your code expects, and you have no reliable way to know how sure the model was.

In September 2026 a company called TypeSafe launched Jev, a model built only for these decisions. It never writes text. You give it a fixed list of possible answers, and it returns one of them with a probability for each. That launch made me want to step back and ask a more basic question. What does a decision in a harness actually need, and how do you check whether a decision maker gives you that on your own traffic?

To find out, I ran three very different gates on the same 1,350 customer messages: a small classifier trained on labelled examples, Claude Haiku 5.5 asked for an answer and a confidence, and Jev. Several people have already tested Jev on other tasks, such as spotting phishing and checking claims against company filings, and I summarize what they found near the end. None of them tested the kind of request that turned out to matter most here, which is one that looks like the agent’s job but isn’t. You don’t need to know anything about calibration or classifiers to follow along. I explain each idea when it first comes up, and there is a short list of terms right after the next section. All the code is in the companion repo, and its scripts reproduce every number in this post.

Here is the short version. All three gates looked fine on the data I tuned them on, and all three made the same confident mistakes on requests that only looked like their job. The fix that worked best was not a better model or a cleverer probability. It was writing down, in one sentence, what the agent does not handle. That sentence has to be exact, though. Claude followed the one I wrote word for word, and because I wrote it too broadly, Claude started sending away requests it should have handled. At the end, I pull all of this together into a guide for when a decision model is the right tool and where it fits in a harness.

What a decision gate is

I’ll call each of these decision points a gate. A gate is a place in the harness where the code has to pick one answer from a fixed list before it can continue. The list might be the skills an agent has, or simply yes and no.

A good gate gives you two things. The first is the answer itself. The second is a probability, which tells you how likely that answer is to be right. The harness then applies a simple rule. If the probability is high enough, it acts on the answer without asking anyone. If it isn’t, it hands the decision to a person or to a stronger model. Handing a decision over like this is called escalating it.

Figure 1. A routing gate in a banking support agent, with two real messages and the probabilities the trained classifier gave them in this experiment. The rule at the threshold decides whether the harness acts alone.

Look at the second message in Figure 1. The gate’s top answer was account_blocked at 47%, but the person was actually reporting fraud. Because the gate was unsure, the harness asked a person instead of running the wrong skill. This is why a probability matters so much, since a gate that knows when it is unsure is far safer than a gate that is always sure.

This only works if the probabilities are honest. If a gate says 95% and is right only 70% of the time, your threshold rule lets through mistakes you thought you had filtered out. Checking that honesty is most of this post.

Terms you’ll see in this post

• Harness: the code around a model that turns it into an agent. It calls the model, runs tools, keeps track of state and decides what happens next.
• Gate: a point in the harness where the code must pick one answer from a fixed list before it continues.
• Decision model: a model built only to answer gate questions. It returns one of your allowed answers with probabilities instead of writing text.
• State: in a request to a decision model, the text or data the decision is about, such as a customer message.
• Typed answer: an answer that is guaranteed to be one of the values you declared, so your code never has to handle anything else.
• Structured output: an API feature that forces a model’s reply to match a JSON schema, for example to pick a skill name from a fixed list.
• Confidence: in this post, the probability a gate gave to the answer it picked.
• Calibration: whether a gate’s confidence can be trusted. A well-calibrated gate that says 80% on a set of requests is right on about 80% of them.
• Reliability diagram: a chart that groups a gate’s answers by confidence and shows how often each group was right, so you can see whether the confidence can be trusted.
• ECE (expected calibration error): the average gap between how confident a gate was and how often it was right. Zero means perfectly calibrated, and lower is better.
• NLL (negative log-likelihood): a score that punishes a gate heavily for being very sure of a wrong answer. Lower is better.
• Threshold: the confidence a gate must reach before the harness acts on its answer without a person.
• Escalate: to hand a decision to a person or to a stronger model because the gate was not sure enough.
• Cascade: a setup where a cheap gate handles the decisions it is sure about and passes the rest to a stronger model or a person.
• Out of scope: a request that none of the agent’s skills is meant to handle.
• Near-miss: an out-of-scope request that looks very much like an in-scope one.
• Validation set: labelled examples kept out of training and used to choose settings, such as the threshold.
• Logprobs: short for log probabilities. Some LLM APIs can return, for each word piece the model writes, how likely the model thought each candidate was.

What a gate needs

I think a gate in a harness needs three properties, and it is worth checking each one separately.

The first is a typed answer. The output must always be one of the values your code knows how to handle. If a gate can return “probably the balance skill, but it could also be transactions”, your code needs a parser, and every parser eventually meets an answer it can’t read.

The second is an honest probability. The confidence the gate reports should match how often it is actually right, so that a threshold means what you think it means. This is the property most people never measure.

The third is that it must be cheap and fast. Gates run on every request and often on every step of an agent’s loop. A gate that takes eight seconds and a few cents is fine once per task, but not twenty times per task.

Four ways to build a gate

There are four common ways to get a decision with a probability attached. They differ mostly in where the probability comes from.

The first approach is the most common, and research suggests it has the weakest probability. Xiong et al. (ICLR 2024) found that LLMs asked to state their confidence tend to be overconfident. The GPT-4 technical report showed a related effect: the pre-trained GPT-4 model was well calibrated on a subset of the MMLU benchmark, but the post-training that turns it into a chat assistant “hurts calibration significantly”. Keep that in mind, because Claude surprised me here.

The second approach reads the probability the model actually computed, instead of a number it wrote. You list the answers as letters, ask for one letter, and look at the probability the model gave each letter. Not every provider exposes these numbers, though. OpenAI’s Chat Completions API returns them for many models, and the OpenAI Cookbook has a guide on using them for classification, but developers on OpenAI’s forum report errors when they request them from some newer GPT-5 models. Anthropic’s Messages API doesn’t return them at all. Local servers such as llama.cpp’s llama-server and vLLM do. I didn’t run this approach for this post. The repo has an implementation for any OpenAI-compatible server if you want to try it with a local model.

The third approach is the oldest. You collect labelled examples of real requests and train a small model on them. In the repo it is a few lines of scikit-learn, which turns each message into word and character counts and fits a logistic regression on top:

# gates/classifier.py
features = make_union(
    TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True),
    TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5), sublinear_tf=True, min_df=2),
)
model = make_pipeline(features, LogisticRegression(C=10, max_iter=2000))
model.fit(texts, labels)
probs = model.predict_proba(["where can i find my routing number"])

For Claude, I used the first approach from the table: ask for an answer and a confidence in the same reply. I couldn’t use logprobs, because Claude’s Messages API doesn’t return the model’s probabilities for each token.

I did use one API feature to make the answer safer. With structured output, you give the API a JSON schema, and Claude’s reply is forced to match it. My schema says the reply must have two fields: skill, which must be one of the 16 allowed names, and confidence, which must be a whole number. That means the skill is always a value my code knows how to handle. The confidence is different. It is still just a number Claude chooses to write, not a probability the model computed, which is why Result 2 checks whether it can be trusted.

# gates/claude.py
SCHEMA = {
    "type": "object",
    "properties": {"skill": {"type": "string", "enum": LABELS},
                   "confidence": {"type": "integer"}},
    "required": ["skill", "confidence"],
    "additionalProperties": False,
}
resp = client.messages.create(
    model="claude-haiku-5-5", max_tokens=200, system=SYSTEM,
    messages=[{"role": "user", "content": verbal_prompt(message)}],
    thinking={"type": "disabled"},
    output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
)

Here is what each part does. LABELS is the list of 15 skill names plus out_of_scope, and the enum line limits skill to those values. "additionalProperties": False stops Claude from adding fields the schema doesn’t name. SYSTEM and verbal_prompt(message) hold the instructions and the customer message, and they are the same prompt the repo gives any LLM. max_tokens=200 caps the length of the reply, which is plenty, since a typical answer used about 21 tokens. output_config turns on structured output with the schema above. The API can’t enforce a range of 0 to 100 for the confidence, so my code checks the range itself and escalates anything outside it.

The thinking line came from a real failure. Claude Haiku 5.5 thinks before it answers by default, and it hides that thinking unless you ask to see it. The thinking still counts toward max_tokens. In my first test run I had set max_tokens to 100, and 7 of 20 answers came back empty or cut off, because Claude had spent most or all of the budget on thinking I couldn’t see. A routing decision doesn’t need step-by-step reasoning, so I turned thinking off and raised the limit to 200. If you use an LLM as a gate, either turn thinking off or leave enough room for it, and treat any answer your code can’t read as an escalation.

How a decision model works

The fourth approach is the new one, so it is worth slowing down for. TypeSafe calls Jev a “System One” model. The name seems to borrow from psychology, where System 1 is fast, intuitive judgment and System 2 is slow, step-by-step reasoning. An LLM that writes out its thinking is closer to System 2. Jev never writes text at all. It only decides.

You send it two things. The first is the state, which is whatever the decision is about, such as a customer message, a tool call or a document. The second is one or more questions about that state. TypeSafe’s SDK has three kinds of question. A Choice picks one of several named options and returns a probability for each one. A Noul is a yes-or-no question and returns the probability of yes. A Score rates the state on an ordered scale, such as low, medium and high priority, and returns a probability for each level. You can ask several questions about the same state in one call.

What comes back is never text. It is the probabilities themselves, so there is nothing to parse and no answer outside the list you declared. You pay only for the input, because output tokens are free. Figure 2 shows the difference on a real message from this experiment.

Figure 2. The same banking message sent to Claude Haiku 5.5 and to Jev, with the real answers each one gave in this experiment. Both picked the right skill, but only the decision model shows how close the second choice was.

Jev’s runner-up in Figure 2 is useful information. “My electric bill should be paid today” could be a request to pay the bill, or a question about when it is due, and Jev gave the second reading a third of its probability. Claude’s single number, 90, can’t tell you that. With the threshold I pick for Jev later in this post, 0.67, this message would go to a person even though Jev’s top answer was right. Whether that is the behaviour you want depends on what a wrong bill payment costs you.

This is the request the repo sends, using TypeSafe’s Python SDK, version 0.7.4:

# gates/jev.py
from typesafe_sdk import Choice, TypeSafeClient
question = Choice(
    instructions="Which skill of a banking support agent should handle this customer message?",
    criteria=descriptions(),  # the same skill descriptions Claude sees
)
resp = TypeSafeClient().system_one(state=message, questions={"skill": question})
answer = resp.answers["skill"]
answer.choice, answer.probabilities, answer.confidence

One detail matters if you ever compare thresholds across tools. Jev’s confidence is not the top probability. TypeSafe’s docs define it as the top probability rescaled so that a uniform guess scores 0 and certainty scores 1, which is (p_max − 1/n) / (1 − 1/n) for n options. With 16 options, a top probability of 0.55 is a confidence of 0.52. In this post I use the top probability for every gate, so the numbers compare like for like.

Decision models have limits, and TypeSafe is open about them. Its page on the weak spots of jev-1.13 says that Jev “is not a calculator” and doesn’t count reliably. It reads dates as text rather than as ordered quantities, and it gets less accurate when the state is full of content unrelated to the decision. It also says Jev “answers the question you wrote, not the one you meant”, and that in a Choice it “leans toward the option that comes first”. The advice is to keep arithmetic, counting and date logic in your own code and to ask Jev only for the judgment. Keep the point about literal reading in mind, because the wording of a question turns out to matter a lot in Result 4.

The lab: a banking agent with 15 skills

To compare gates, you need a task with known right answers. I used CLINC150, a public dataset of short user requests published by Larson et al. at EMNLP 2019. It has 150 request types, called intents, across ten areas such as banking, travel, and kitchen and dining. It also has a set of requests that don’t match any intent, which makes it a good stand-in for real traffic.

The agent in my lab is a banking support agent with 15 skills, one per CLINC150 banking intent, such as transfer, routing and report_fraud. For every message, the gate must pick one of those skills or answer out_of_scope, which sends the message to a person.

The test set mixes three kinds of traffic, with 450 messages of each kind:

• In-scope: banking requests that one of the 15 skills should handle.
• Near-miss: requests from CLINC150’s credit card area, such as questions about APR or card limits. They sound like banking requests, but no skill handles them, so the right answer is out_of_scope. None of them appear in training or validation.
• Unrelated: requests from CLINC150’s out-of-scope set, such as “how many prime numbers are there between 0 and 100”.

The near-miss group is the important one. Every agent eventually meets requests that look like its job but aren’t, and they rarely show up in the data you prepared before launch. The idea of testing with these look-alikes isn’t mine. Zhang et al. (2022) showed that intent classifiers built on pretrained transformers handle these “in-domain out-of-scope” requests much worse than unrelated ones, and they built a banking and credit card version of CLINC150 to show it. My version is simpler. Instead of holding out a few intents inside each area, I hold out the whole credit card area.

The first script downloads CLINC150 and builds the splits:

$ python3 data/prepare.py
train  2200 rows  {'in': 1500, 'near': 0, 'far': 700}
val     640 rows  {'in': 300, 'near': 0, 'far': 340}
test   1350 rows  {'in': 450, 'near': 450, 'far': 450}

In the repo’s data, far means unrelated and near means near-miss. The classifier learns from the training split, which has 100 labelled examples per skill and 700 out-of-scope examples. Claude and Jev never see that data. They only get one short description per skill:

# gates/common.py (excerpt)
SKILLS = {
    "interest_rate": "Tell the user the interest rate on their account",
    "report_fraud": "Report a fraudulent charge on a bank account",
    "freeze_account": "Freeze or lock the user's bank account right away",
    ...
}
OOS_DESCRIPTION = "None of the skills above handles this request"

One caveat applies to everything that follows. CLINC150 has been public since 2019, so Claude and Jev may have seen it, or text very like it, during training. An independent study by scienthoon found Jev’s calibration error was 0.024 to 0.032 on public benchmarks but 0.107 on synthetic support tickets it could not have seen. The author points out that the public benchmarks are probably in Jev’s training data, though nobody outside TypeSafe can check. The same could be true of CLINC150, and that could make both models look better here than they would on your own traffic. It is one more reason to rerun this kind of test on real requests from your agent.

Result 1: accuracy looks fine for all three

Each gate answered the same 640 validation and 1,350 test messages. evaluate.py scores them. Here is its output for Jev, run from my laptop:

$ ./run_api_arms.sh --full
...
== jev (1350 test requests)
accuracy        all 0.922 | in-scope 0.933 | near-miss 0.840 | unrelated 0.993
latency         p50 86.59 ms | p95 147.82 ms
tokens/request  in 635 | out 149
calibration     ECE 0.044 | NLL 0.820 | mean confidence 0.956
gate threshold  0.67 (picked on validation for at most 2% wrong)
  validation          handled  93.7% of in-scope | wrong  1.4% (4 of 285 automatic actions)
  test, no near-miss  handled  91.1% of in-scope | wrong  2.1% (9 of 419 automatic actions)
  test, all           handled  91.1% of in-scope | wrong 14.0% (67 of 477 automatic actions)

The accuracy line is the one most people stop at. Here it is for all three gates:

All three are in the low nineties, and Jev is slightly ahead. Every gate handled unrelated requests well. Every gate also sent between 16% and 21% of the near-miss requests to a banking skill, which is the first sign of trouble. The rest of the output is about whether you can trust these answers, and that is where it gets interesting.

Result 2: are the probabilities honest?

A probability is only useful if you can take it at face value. Think of a weather forecaster. If she says there is a 70% chance of rain on many different days, it should rain on about 7 out of 10 of those days. If it rains on all of them, she is underselling her forecasts. If it rains on only 4 of them, she is overselling them. A gate’s confidence works the same way. A gate whose confidence matches how often it is right is called well calibrated.

To check this, you sort a gate’s answers into groups by the confidence it gave, such as 60 to 70% or 90 to 100%. Then you count how often the answers in each group were right. A chart of these groups is called a reliability diagram. Figure 3 shows one for the trained classifier, and here is how to read it:

• Each blue bar is one group of answers, and its height shows how often those answers were right.
• The dashed diagonal shows a perfectly honest gate, where 70% confidence means right 70% of the time.
• A bar above the line means the gate was right more often than it claimed, so it was too modest. A bar below the line means it was wrong more often than it claimed, so it was overconfident. Overconfidence is the dangerous one, because it lets mistakes slip past your threshold.
• The grey strip underneath shows how many answers fell in each group. A bar far from the line matters much less when its group holds only a handful of answers.

Figure 3. Reliability diagrams for the trained classifier on the test set, before and after temperature scaling, which I explain below. The closer the blue bars are to the dashed line, the more honest the confidence.

As trained, the classifier was too modest. Every bar sits above the line. On average it gave its answers 80% confidence, but it was right 90.5% of the time.

Charts are hard to compare across gates, so it helps to turn each one into a single number. The usual one is the expected calibration error, or ECE. It is the average gap between how confident the gate was and how often it was right, with bigger groups counting for more. Zero means perfectly honest, and lower is always better. You can read it roughly as percentage points, so the classifier’s ECE of 0.102 means its confidence was off by about 10 points on average. If you write code, this is how the repo computes it:

# gates/common.py (simplified)
def ece(rows, bins=10):
    scored = [(top_prob(r), r["pred"] == r["label"]) for r in rows]
    total = 0.0
    for b in range(bins):
        inside = [(p, ok) for p, ok in scored if b / bins < p <= (b + 1) / bins]
        if inside:
            conf = sum(p for p, _ in inside) / len(inside)
            acc = sum(ok for _, ok in inside) / len(inside)
            total += len(inside) / len(scored) * abs(conf - acc)
    return total

There is a standard fix for a gate that is too modest or too confident, called temperature scaling (Guo et al., ICML 2017). Think of it as one dial for the whole gate. Turning it one way makes every answer more confident, and turning it the other way makes every answer less confident. It never changes which answer the gate picks, so accuracy stays exactly the same. You set the dial using the validation set. In code, the dial is a number called T. Every probability is raised to the power 1/T and then rescaled so the probabilities add up to one again:

# evaluate.py
def rescale(rows, temperature):
    for r in rows:
        logits = {k: math.log(max(p, 1e-12)) / temperature for k, p in r["probs"].items()}
        ...  # softmax over the logits again

For the classifier, the best setting was a T of 0.61, which made it more confident. Its calibration error fell from 0.102 to 0.024, or about 2 points, and the bars in the right panel of Figure 3 sit close to the line.

Now the same check for all three gates. Figure 4 uses the same kind of chart, and the table after it puts the key numbers side by side.

Figure 4. Reliability diagrams for all three gates on the same 1,350 test messages. Check the grey strips as well as the bars. Most of Claude’s and Jev’s answers sit in the most confident group, so the bars on the left of their charts hold very few answers.

This is where Claude surprised me. Its stated confidence was the most honest of the three, off by about 3 points on average, which goes against the research I quoted earlier. On average it claimed 92% and was right 90.7% of the time. I wouldn’t read too much into one model on one task, but it is a good reminder to measure instead of assuming. Claude’s confidence does have a practical limit, though. It used only 22 different values. Nearly two thirds of its answers were 95 or higher, and most of the rest landed on a few round steps such as 85 and 90. That leaves you few places to put a threshold, and Result 4 shows what that can cost.

Jev’s calibration error looks good too, at 0.044, or about 4 points. But 87% of its answers fell in the most confident group, and when it was wrong, it was often very sure. Of its mistakes, 42% came with 90% confidence or more, against 27% for Claude and 6% for the classifier. ECE is an average over all the answers, so a small number of very confident mistakes barely moves it.

That is what the last column of the table measures. NLL, short for negative log-likelihood, looks at the probability the gate gave to the right answer and charges a penalty that grows very fast as that probability gets close to zero. If the gate gave the right answer 50%, the penalty is 0.7. If it gave the right answer only 1%, because it was 99% sure of a wrong one, the penalty is 4.6. The score is the average penalty over all answers, and lower is better. Jev’s NLL was 0.82, much higher than Claude’s 0.48 and the classifier’s 0.41, because of those very confident mistakes. The next result shows why they matter.

Result 3: the threshold that didn’t hold

Honest probabilities matter because of what the harness does with them. Remember the rule from Figure 1: if the gate’s confidence is above a threshold, the harness acts on its own, and otherwise it hands the request to a person. In code, the rule is just this:

# gates/common.py
def gate(row, threshold):
    if row["pred"] == "out_of_scope":
        return "escalate"
    if top_prob(row) < threshold:
        return "escalate"
    return "automate"

Choosing a threshold is a trade-off. A low threshold lets the harness handle more requests on its own but lets more mistakes through. A high threshold catches more mistakes but sends more work to people. I set the same goal for every gate: no more than 2 in every 100 automatic actions should run the wrong skill.

The test then has two steps. First, for each gate I picked the lowest threshold that met the 2% goal on the validation set, which is the data I was allowed to tune on. Second, I checked whether that promise still held on the test set, which played no part in any choice. I checked it twice. The first check used only familiar requests, meaning banking and unrelated ones like those in validation. The second used all the requests, including the near-miss credit card requests that validation never had.

The pattern is the same for all three gates. On familiar requests, they kept the promise and stayed at about 2% or below. Once the look-alike credit card requests arrived, wrong actions jumped to between 9% and 14%, which is five to seven times the goal. Almost all of the extra mistakes were those look-alikes: 32 of the classifier’s 39 wrong actions, 45 of Claude’s 50 and 58 of Jev’s 67.

Figure 5 shows the same thing for the classifier at every possible threshold. Read it from left to right. As the threshold rises, the gate acts alone less often, and fewer of its actions are wrong. The blue and green lines, which are validation and the familiar test requests, reach the 2% target close to the threshold of 0.55. The orange line, which includes the look-alikes, is still at 9.3% there, and it only gets under 2% at a threshold of about 0.85.

Figure 5. Wrong automatic actions for the trained classifier at every threshold. The threshold picked on validation keeps its promise on familiar requests and breaks once look-alike requests arrive.

Here is what each gate did with five of the look-alike messages. Every one of them should have gone to a person. A skill name means the harness ran that skill on its own, with the gate’s confidence in brackets, and “escalated” means a person got the message.

Claude and Jev agreed with each other almost every time. Both turned an APR question into an interest-rate question and a declined card into a blocked account, usually with high confidence, and Jev was often completely sure. The classifier escalated many of these messages, because words like “credit card” and “declined” never appeared in its training data. It made its own confident mistakes instead, on short messages such as “what is my apr rate” that look just like its banking examples.

I don’t think the models were stupid here. Read the skill description again: interest_rate tells the user “the interest rate on their account”. Nothing in the task says a credit card account doesn’t count. A sensible person reading that list could make the same call. The classifier had never seen a credit card request, and the descriptions Claude and Jev got never said credit cards were out of scope. In different ways, all three gates were confidently answering a question the task never defined. The scienthoon study I mentioned earlier found the same pattern for Jev on synthetic support tickets. It was most overconfident on a priority label that depended on a policy the ticket text never stated.

Temperature scaling made things worse for the classifier. With its dial set on the validation set, the classifier picked a threshold of 0.63. It then ran the wrong skill on 13.2% of its automatic actions on the full test set, up from 9.3%. The dial was set using validation data that had no look-alike requests in it, so it simply made the gate more confident everywhere, including on the requests it should have doubted.

The lesson I took from this is that a gate’s honesty depends on the traffic you measure it on. A reliability diagram tells you how honest the gate was on the data you checked. It can’t tell you how the gate will behave on requests that look different, so the requests you test with need to include the awkward ones.

Result 4: writing down the scope, and what it cost

If the problem is a gap in what the task says, the fix is to close the gap. How you do that depends on the gate.

For the classifier, the fix is data. I added 10 labelled examples of each credit card intent, 150 messages in total, to the training set as out_of_scope, and 60 more to the validation set. None of them overlap with the test messages.

For Claude and Jev, the fix is one sentence. I changed only the description of out_of_scope:

# gates/common.py
OOS_DESCRIPTION_SCOPED = (
    "None of the skills above handles this request. The agent covers bank accounts and bills only, "
    "so anything about a credit card (its APR, limit, rewards, a lost, damaged or declined card, "
    "or a card application) is out of scope"
)

Then I ran every gate again on the same test messages. Each threshold was picked again on validation data that now included 60 near-miss examples, so the threshold could see the new kind of traffic. For the classifier, python3 -m gates.classifier --with-near retrains it with the extra examples. For Claude and Jev, the --extra flag of the API script runs both gates with the new sentence. It skips any run whose results are already saved, which helped when Jev’s API failed partway through one of my runs and I had to start again. Here is the end of its output for Claude:

$ ./run_api_arms.sh --extra
...
== claude_scoped (1350 test requests)
accuracy        all 0.965 | in-scope 0.907 | near-miss 0.989 | unrelated 1.000
latency         p50 889.14 ms | p95 1827.84 ms
tokens/request  in 843 | out 21
calibration     ECE 0.033 | NLL 0.220 | mean confidence 0.934
gate threshold  0.86 (picked on validation for at most 2% wrong)
  validation          handled  72.0% of in-scope | wrong  0.5% (1 of 217 automatic actions)
  test, no near-miss  handled  74.4% of in-scope | wrong  1.5% (5 of 340 automatic actions)
  test, all           handled  74.4% of in-scope | wrong  1.8% (6 of 341 automatic actions)

Compare it with Claude’s first run in the earlier tables. Near-miss accuracy rose from 0.793 to 0.989, and the last line now meets the 2% target on all traffic. Two other numbers moved the wrong way, though. In-scope accuracy fell from 0.931 to 0.907, and the threshold rose from 0.81 to 0.86. Both come up again below.

A fix has two sides, so Figure 6 shows both for all three gates. The left panel is safety, meaning how many automatic actions were wrong. The right panel is cost, meaning how many banking requests the gate still handled without a person.

Figure 6. Each gate before and after its fix, on all test traffic at the threshold picked on validation. A fix is only good if it improves the left panel without taking too much from the right one.

All three fixes worked on the safety side. The single sentence cut Jev’s wrong automatic actions from 14.0% to 2.2% and Claude’s from 11.5% to 1.8%, so both ended up close to the 2% target. Jev now escalated 94.7% of the near-miss requests, up from 84.0%, and Claude escalated 98.9%, up from 79.3%. The classifier’s 150 labels took it from 9.3% to 3.9%, and it escalated 98.2% of the near-miss requests.

The probabilities became more honest as well. Jev’s calibration error fell from 0.044 to 0.010, and its NLL fell from 0.82 to 0.19, which means it was no longer so sure of itself when it was wrong. Claude’s NLL fell from 0.48 to 0.22, while its calibration error stayed about the same at 0.033.

The fix didn’t always change the answer. With the new sentence, Jev still mapped “what is my apr rate” to interest_rate, but its confidence dropped from 1.00 to 0.54, well under its new threshold of 0.72. The gate became unsure, and the threshold did the rest.

The right panel is where the fixes differ. The classifier and Jev handled about as many banking requests on their own as before. Claude dropped from 85.6% to 74.4%, which means a person now had to look at 11 more of every 100 banking requests. That cost had two causes. Both are worth understanding, because neither of them is specific to Claude.

Cause 1: Claude did exactly what my sentence said

Read the sentence again. It says that “anything about a credit card” is out of scope. CLINC150 doesn’t draw the line there. Its banking intents include requests such as “what were my last 10 charges on my credit card”, which is labelled transactions, and “i would like to report fraudulent activity on my chase card”, which is labelled report_fraud. Of the 450 banking requests in the test set, 52 mention a card or a card brand such as Visa.

Claude took my sentence literally. With it, Claude refused 18 of those 52 requests, compared with 3 before. Jev read the same sentence much more loosely and refused only 4, compared with 2 before. The classifier, which never saw the sentence, refused 5. Here are four of the requests Claude newly sent to a person:

Jev kept the right skill on these requests, but its confidence dropped compared with its first run, where it gave all four a probability of 0.99 or 1.00. On the last one, it fell under Jev’s threshold of 0.72, so a person got that message too. It looks as if Jev noticed that the request and the sentence pulled in different directions, and it showed that as doubt instead of a refusal.

By the dataset’s labels, these are Claude’s mistakes. By my instructions, they are not. I told Claude that anything about a credit card was out of scope, and it obeyed. The mistake was mine. I wrote the sentence from the near-miss errors and never checked it against the banking examples the agent was supposed to keep handling. I can’t tell from the outside why Jev gave the sentence less weight, and I expected the opposite, since TypeSafe’s own docs warn that Jev reads questions literally. On this dataset its reading happened to match the labels better, but if your policy really did exclude every card request, Claude’s literal reading is the one you would want.

A tighter sentence would separate the card account itself from the banking actions that happen to involve a card. For example, it could say: “Questions about a credit card account itself, such as its APR, limit, rewards, replacement or an application, are out of scope. Charges, fraud and bills on a card still go to the matching skill.” I haven’t run this version, so I can’t tell you what it would score.

Cause 2: Claude’s confidence comes in steps

The refusals explain only about 2 points of the drop. At the old threshold of 0.81, Claude with the sentence would have handled 83.6% of banking requests, with 2.3% of its automatic actions wrong. The other 9 points came from the threshold itself, which rose from 0.81 to 0.86.

The threshold rose because of two extra mistakes on validation. With the sentence, Claude made 7 wrong automatic actions at a threshold of 0.81 instead of 5, which put it at 2.8%, over the 2% target. Six of those 7 mistakes had a confidence of exactly 0.85. One of them sent “show all purchases of video games” to spending_history instead of transactions, which is an easy mix-up for a person too. To get under 2%, the threshold had to move just above that step, to 0.86.

Figure 7. Claude writes its confidence as a round number, so its answers pile up on a few values. Moving the threshold from 0.81 to 0.86 escalated every answer at 85%, even though all 41 of them were right.

Figure 7 shows why such a small move had such a large effect. Claude writes its confidence as a round number, so its answers pile up on a few values instead of spreading out. Of the 408 banking requests it routed correctly, 41 had a confidence of exactly 85%. Raising the threshold past 85 sent all 41 of them to a person at once, which is 9% of all banking requests. For comparison, the same move from 0.81 to 0.86 would have cost Jev, whose probabilities are spread much more smoothly, only 2 points of banking requests.

There are two lessons in this. A validation set of a few hundred messages is small enough that two mistakes can move your threshold. And when confidence comes in coarse steps, one small move can escalate a whole block of answers. If you use stated confidence, look at how its values are spread before you pick a threshold, and check how much traffic you would lose one step on either side of it.

What the fixes do and don’t show

I should be honest about one more thing. I wrote that sentence after reading the mistakes, just as I labelled the 150 examples after seeing them. Both fixes measure how cheap it is to repair a gate once you know about a gap. They don’t measure how a gate handles a gap nobody has noticed yet. In production, you find those gaps by reading the requests your gate escalates or gets wrong, which is a good reason to log every one of them.

The amount of labelled data matters for the classifier in the other direction too. With only 10 examples per skill instead of 100, it sent just 76.2% of banking requests to the right skill. It was also badly underconfident, with an ECE of 0.266, and at its validation threshold it handled only 61% of banking requests on its own. If you can’t label at least a few dozen real examples per answer, a gate that works from descriptions becomes much more attractive.

Does Jev give the same answer twice?

During my first small test runs, Jev’s calibration error on the same 20 messages came out as 0.111, then 0.054, then 0.102. That made me wonder whether its answers were stable, so I asked it all 1,350 test questions a second time. The output of scripts/repeat_check.py:

$ python scripts/repeat_check.py jev jev_repeat
1350 requests asked twice
answer changed      5 (0.4%)
confidence moved    median 0.000 | mean 0.005 | max 0.140
moved more than 0.05  33 (2.4%)

Only 5 answers changed, and every one of them was a near-tie with a top probability between 0.34 and 0.54. A threshold of 0.67 sends all of those to a person anyway. The jumps in my small runs were just what happens when you measure calibration on 20 messages. A reliability diagram needs hundreds of answers before it means anything. I didn’t repeat the Claude run, so I can’t say how stable its stated confidence is.

Latency and cost

I measured latency one request at a time from my laptop in Toronto, over a home internet connection. Costs use each provider’s published price.

Claude Haiku 5.5 costs $0.10 per million input tokens and $0.50 per million output tokens for prompts under 100,000 tokens. Jev’s launch price is $0.042 per million input tokens, and its output tokens are free. On this task, Jev was about 3 times cheaper and about 11 times faster than Haiku 5.5. TypeSafe’s headline numbers, 193.6 times faster and 444.6 times cheaper, come from its own workflows and compare against frontier models. Vals AI checked them independently and found the cost claim holds against a frontier model, at about 1/498 of the price, and the speed claim holds when many questions are asked about one document in a single call. Against a small, cheap model, the gap is much smaller, though still large for something that runs on every step.

Latency matters more than it looks. A gate runs on every request, and in an agent loop it may run at every step. Adding one second to each step of a twenty-step task adds twenty seconds before the model has done any real work. At that scale, a gate under 100 milliseconds is a real difference.

What this test can’t tell you

This is one task on one dataset, so it is worth being clear about what it doesn’t show. CLINC150 messages are short, clean and written by crowd workers, while real customer messages are longer and messier. As I said earlier, the models may also have seen the dataset during training. I ran Claude only once per setup, so I can’t say how much its numbers would move on a second run. In the Jev request, out_of_scope was the last of 16 options, and TypeSafe’s docs say Jev leans toward options that come first, so a different order might change its near-miss results. I didn’t test that. Finally, both fixes were written after I had seen the mistakes, as I explained in Result 4.

What other people found

I’m not the first to test Jev independently. Within a month of its launch, several people published their own tests on very different tasks, and they are worth reading alongside this one. Their results and mine mostly agree, and where they seem to differ, the difference teaches something.

The clearest shared finding is that calibration depends on the task. Vals AI compared Jev with 11 other systems on 400 claims checked against companies’ annual reports, and on a slice of the LegalBench benchmark. Jev’s calibration error was 0.011 on the claims, the best of the twelve, but 0.108 on LegalBench, where it was also the least accurate. Paul Ravvich tested spam detection and the Banking77 intent dataset and found Jev no better calibrated than two GPT-6 models. The scienthoon study measured 0.024 to 0.032 on public benchmarks and 0.107 on fresh synthetic tickets. My 0.044 sits between those numbers. In every case, the calibration number belongs to a task, so it has to be measured on yours.

Jev also seems to be most overconfident when the right answer depends on something the question doesn’t say. In scienthoon’s tickets, the priority label depended on a policy that wasn’t in the text, and Jev was right on only 44.7% of them while giving its answer 0.74 probability on average. A pre-registered study by PriorBench found that without a “none of these” option, Jev didn’t flag any of 30 out-of-scope messages, and its wrong answers came with 0.99 confidence. My near-miss results fit the same pattern. Even with an explicit out_of_scope answer, Jev was very sure about credit card requests until the description said what out of scope meant.

Ravvich found that Jev’s confidence dropped to about 0.66 to 0.69 on its mistakes, while a cheap GPT-6 model stayed at 0.82 to 0.85. That looks like it contradicts my Result 2, where Jev was often very sure when it was wrong, but the two results fit together. On ordinary banking requests, Jev’s average confidence on its mistakes in my test was 0.73, close to Ravvich’s numbers. Most of its very confident mistakes, 33 of the 44 at 0.9 or higher, were near-miss requests. Jev seems to know when it is torn between two of its own answers, but not when a request isn’t its job at all.

How you ask matters as much as which model you ask. In jev-phishing-bench by the GitHub user anisselbd, Jev, asked the single question “is this phishing?”, was right on 62.6% of 2,000 emails, against 81.3% for Claude Haiku 4.5. When the same call also asked five simpler questions, such as whether a link pointed to free hosting, and a small logistic regression combined the answers, accuracy rose to 95.1%. PriorBench found that wrong descriptions of the options are “catastrophic”, pushing accuracy below random guessing, while leaving the descriptions out entirely cost less than a point. Annabell Schäfer and Hassieb Pakzad’s Langfuse guide to using Jev for evaluations recommends always giving it an “unknown” or “needs review” answer, so it doesn’t have to pick the least wrong one. My scope sentence is one more example of the same lesson.

When you have labelled data, a small trained model is a strong baseline. In a comment on the phishing benchmark, someone fine-tuned a 4-billion-parameter Qwen model on 1,000 of the emails in 18 minutes and reached 97.4% accuracy with a calibration error of 0.010. Carson Rodrigues and Oysturn Vas compared Claude Haiku with fine-tuned classifiers on the full CLINC150 dataset and found Claude statistically tied with a fine-tuned RoBERTa model. Their median latency for Claude, 981 ms, is almost exactly the 978 ms I measured, which is a nice independent check.

TypeSafe launched Jev on September 15, 2026, and it is still in early access. Its launch post reports end-to-end latency of 70 to 500 milliseconds, which matches what I and others measured. The company also says its workflow evaluations were built by its own team, that the reference answers came from averaging two frontier models, and that the headline numbers are “on the higher end” of what users should expect. As far as I can tell, TypeSafe hasn’t published calibration error numbers for Jev itself, which is why these independent tests matter. One more detail is worth knowing. TypeSafe’s open-source System One Adapter, which wraps an LLM so you can compare it with Jev, gets the LLM’s probabilities by asking the model to write them in JSON. That is the same stated-confidence approach I used for Claude, so its comparisons and mine measure the same kind of LLM baseline.

If you want one place to start, Rajesh Beri’s write-up collects several of these studies and pointed me to some of them.

When to use a decision model

On this task, Jev was the most accurate gate, and it was the faster and cheaper of the two hosted models. Its weakness was the one Result 2 hinted at: on requests outside its job, it was often completely sure. With one sentence of scope, it improved the most of the three gates and still handled 91% of banking requests on its own, while the same sentence cost Claude 11 points. Part of that is because Jev read my imperfect sentence more loosely than Claude did, so a better sentence might narrow the gap.

Putting my results together with everyone else’s, this is how I would choose a gate:

As for where a decision model fits in a harness, the natural places are the gates from Figure 1. These include routing a request to a skill, checking whether a tool call is safe, deciding whether a task is finished, flagging a reply for review, and choosing the next sub-agent. Sydney Runkle and Hunter Lovell at LangChain have published middleware that uses Jev for two of these, picking a model for each request and blocking risky tool calls before they run. What these places have in common is that the decision is small, frequent and bounded, which is exactly what a decision model is built for.

Whichever gate you choose, start it in shadow mode. Run it next to whatever makes the decision today, on a thousand or more real decisions that include awkward requests, and compare its answers with the right ones before you let it act alone. Pin the model version, because a new version can move both the answers and the probabilities. Beri’s write-up makes the same recommendation, and the checklist below covers the rest.

My checklist for a gate

Before I trust any gate in a harness, whether it is a classifier, an LLM call or a decision model, I check these things on my own traffic:

1. The answer is always one of the allowed values. An answer the code can’t read, or a request that fails, is treated as “escalate”. Jev’s API failed on one of my requests during a run, so this is not hypothetical.
2. One of the allowed answers is a way out, such as out_of_scope, “none of these” or “needs review”, so the gate never has to pick the least wrong option.
3. The gate’s description says what the agent does not handle, not only what it does. Before I ship that sentence, I run it against labelled in-scope examples to make sure it doesn’t push away requests the agent should keep.
4. I measure calibration with a reliability diagram and ECE on hundreds of answers, and I check NLL to catch a gate that is very sure when it is wrong.
5. The threshold is picked on a validation set for a target error rate, and then confirmed on separate test data.
6. I look at how the confidence values are spread. If they come in a few coarse steps, I check how much traffic moves when the threshold shifts one step either way.
7. The test data includes near-miss requests that look in scope but aren’t, and I report results with and without them.
8. I log every escalation and read some of them, because that is where the next gap in the description shows up.
9. Latency and cost are measured per decision and multiplied by how many times the harness calls the gate per task.

None of this depends on which kind of gate you choose, and that is the main point of this post. A decision model like Jev may well be the right tool for many gates, and on this task it did very well once the task was described properly. You only find that out by running it on the traffic your agent will actually see, including the strange requests at the edges.

Next in this series I’ll look at benchmarking best practices: how much noise, hardware limits and repeated runs move agent benchmark scores.

References

• Companion code for this post: github.com/alireza1989/harness-decision-gates
• TypeSafe, Introducing System One Models & Jev: typesafe.ai/blog/introducing-system-one-models-and-jev
• TypeSafe documentation, Quickstart, Confidence, System One models and jev-1.13 weak spots
• TypeSafe, System One Adapter (MIT license): github.com/typesafe-ai/system-one-adapter-python
• Let’s Data Science, TypeSafe AI Launches Jev for Structured Software Decisions (September 19, 2026): letsdatascience.com
• Anthropic, Messages API reference: platform.claude.com/docs/en/api/messages
• Anthropic, pricing: platform.claude.com/docs/en/about-claude/pricing
• Larson et al., An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction, EMNLP-IJCNLP 2019 (CLINC150, CC BY 3.0): aclanthology.org/D19-1131, data at github.com/clinc/oos-eval
• Zhang, Hashimoto, Wan, Liu, Liu, Xiong and Yu, Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection, NLP4ConvAI 2022 (the idea of in-domain out-of-scope requests): aclanthology.org/2022.nlp4convai-1.2
• Vals AI (Connor Frank), An Independent Evaluation of TypeSafe’s Jev (October 6, 2026): vals.ai
• Paul Ravvich, TypeSafe’s Jev beyond the hype: an Independent Benchmark (September 2026): medium.com
• scienthoon, jev-ood-calibration (September 19, 2026): github.com/scienthoon/jev-ood-calibration
• PriorBench, pre-registered evaluation of Jev (September 20, 2026): github.com/priorbench/jev
• anisselbd, jev-phishing-bench (September 17, 2026): github.com/anisselbd/jev-phishing-bench, and the fine-tuned Qwen baseline in issue #1
• Annabell Schäfer and Hassieb Pakzad, Using TypeSafe’s Jev for evals, Langfuse (September 18, 2026): langfuse.com
• Rajesh Beri, TypeSafe’s Jev Scores 62.6% Asked Once and 95% Split Five Ways (September 20, 2026): beri.net
• Sydney Runkle and Hunter Lovell, Building a harness with Jev, LangChain (September 17, 2026): langchain.com
• Carson Rodrigues and Oysturn Vas, When Do LLMs Replace Fine-Tuned NLU? A Decision Framework for Intent Detection in Production Conversational Systems (2026): arxiv.org/abs/2608.20371
• Guo, Pleiss, Sun and Weinberger, On Calibration of Modern Neural Networks, ICML 2017 (temperature scaling and ECE): arxiv.org/abs/1706.04599
• Xiong et al., Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs, ICLR 2024: arxiv.org/abs/2306.13063
• OpenAI, GPT-4 Technical Report (2023), Figure 8 on calibration before and after post-training: arxiv.org/abs/2303.08774
• OpenAI Cookbook, Using logprobs: developers.openai.com/cookbook/examples/using_logprobs
• OpenAI Developer Community, GPT 5.2 LogProbs support removed? (March 2026): community.openai.com
• llama.cpp server documentation: github.com/ggml-org/llama.cpp