We benchmarked TypeSafe's Jev against four OpenAI models

Before the numbers, what you should know while reading them: we sell an AI helpdesk, the classification job in this test is one our own product runs on live customer mail, and we would have been happy to find a cheaper model. Read the numbers rather than our framing of them.

TypeSafe came out of stealth on 15 September with a model called Jev. It does not generate text. You send it a state and a set of typed questions, and it returns an answer per question with a probability distribution and a confidence score. Not prose, not code, not a two word label. That restriction is the entire product, and it is priced accordingly at four cents per million input tokens with output free.

We run four model calls in production that produce no prose at all. They decide things. So we tested it.

What we tested#

When our AI needs something only a third party knows, such as where a parcel actually is, it emails the carrier and parks the customer conversation. When the carrier replies, something has to decide what that reply was: ANSWER, ACKNOWLEDGEMENT, UNRELATED, or UNCLEAR, where UNCLEAR routes the whole thing to a human.

Five models. Jev 1.13, gpt-4o-mini, gpt-5.6-luna, gpt-5-nano and gpt-oss-20b. Forty three cases, five repetitions each, two prompt conditions for the chat models, run through one code path so the columns mean the same thing. About 2,400 calls in total and 2.30 dollars.

Twenty six of the cases are our production regression suite. Seventeen are held out: written before any model ran against them, in three labelled groups, including cases in German, Hungarian and Slovene, and three adversarial ones.

Four bugs in our own benchmark#

This is the part we think is worth your time, because every one of these produces a confident table that is wrong, and nothing in the output looks wrong.

One. Reasoning models were being truncated before they answered. We set max_tokens to 200, which is generous for a one word verdict. gpt-5-nano spent all 200 on internal reasoning, returned finish_reason "length" with null content, and our parser turned that into UNCLEAR. It scored 29 out of 129. That number was not a model result, it was our configuration. At max_tokens 4000 the same model scores 42 out of 43.

The subtle part is why the parser did that. In production, unparseable model output should fail safe and fetch a human, so UNCLEAR is exactly right. In a benchmark it is fatal, because "the model never answered" and "the model answered UNCLEAR" become the same row. Three of our four chat models reason invisibly.

Two. Our own network errors were being scored as model failures. A later run recorded 1,445 invalid rows. All of them were fetch failures from running two jobs at once on one laptop. The account was fine. We now retry transport faults and record them as a separate outcome that is excluded from accuracy entirely, because it is a fact about our laptop and not about anybody's model.

That one had a nastier consequence. The same network fault hit our label audit and killed the last eight cases for all three labellers at once. Contiguous missing data is worse than noisy data, because it looks like a clean result: we got an agreement score of 0.885 that had silently dropped every adversarial case. With the gap closed it is 0.794. The first number was not a lie, it was an average over the easy two thirds.

Three and four came from someone else. We asked an independent reviewer to attack the design. It found that our accuracy table pooled the tuned and held-out fixtures into one number, which moves with the ratio of set sizes rather than with model quality, and that our confidence intervals treated five repetitions of one case as five independent emails.

That second one mattered most. Repeated calls on the same case are clustered. Perfect held-out performance printed as an interval of 95.7 to 100 per cent when the defensible case-level interval is 81.6 to 100. We would have published the narrow one, and every comparison in the post would have looked far more decisive than the evidence supports.

So the tables below report two things separately. A call rate, which is descriptive and carries no interval. And a case rate, which is the fraction of cases a model got right on a majority of its runs, over 17 or 26 cases, which is the number that generalises and carries the interval.

Accuracy: a tie, and not a close-run one#

Every pairwise comparison came back null. Discordant counts of nought to two, and after correcting for multiple comparisons every single adjusted p-value was 1.0. The intervals overlap almost entirely.

The honest conclusion is that forty three cases cannot tell these models apart. We could have written "Jev matches frontier models at a fraction of the price" from the top row of that table, and it would have been a sentence with nothing behind it.

Two things did show up. Our production regression suite is saturated: gpt-4o-mini scored 130 out of 130 on it, which is what you would expect from a suite the prompt was tuned against, and it is why we built the held-out set at all. And the adversarial cases split the field, with Jev and Luna at 10 of 15 against 5 of 15 for gpt-4o-mini and gpt-oss-20b. That is three cases though, so treat it as a hint.

The case that beat everything was a carrier replying with a single word, "Confirmed." Our rule says that is UNCLEAR, because you cannot tell what was confirmed. Several models call it an ANSWER, consistently, at low confidence.

Cost: the price page will mislead you#

Measured from the provider's own usage figures, serially, including reasoning tokens.

gpt-5-nano has the cheapest list price in this set at five cents per million input tokens, and it was the most expensive model we ran. Five and a half times Jev, twice gpt-4o-mini. It burns about 509 reasoning tokens on a four way classification. Those tokens are billed and you never see them. Anyone picking a classifier off a pricing page gets this exactly backwards.

Jev is genuinely the cheapest, and the margin is much smaller than the launch coverage suggested. Against gpt-oss-20b it is 1.6 times cheaper. The very large multiples quoted at launch assume a baseline that reasons at length before answering. On a tight classification prompt most of that advantage is already spent.

One more trap, found while writing this. Running calls concurrently made gpt-5.6-luna look 1.5 times cheaper than it is, because identical prompts fired in parallel hit the prompt cache. Every cost figure above is from a serial run. If you benchmark cost under concurrency you will under-report it.

The cost advantage reverses on longer inputs#

Free output only helps while output is a real share of the bill. Jev's INPUT price, four and a bit cents per million, is actually higher than gpt-oss-20b's three cents. So the longer the email, the more input dominates, and the cheaper input price wins.

Break-even is at about 3,400 input tokens. A long carrier email with a quoted thread underneath it sits right in that zone. So a bigger payload does not make this class of model cheaper, it makes it worse, and that is the opposite of the intuition most people bring to "output is free".

And at real volume, none of it matters#

This is the part that should probably have come first.

We pulled our own production numbers: 1,995 inbound customer messages in the last thirty days. That is the highest volume classification path we have.

At our real volume, switching classification model saves under two dollars a year. We would need roughly a hundred times our current traffic before the saving paid for a single day of engineering time.

The benchmark you are reading cost 2.30 dollars to run, which is more than a year of the spending it was evaluating.

We are not embarrassed by that, because the latency and reasoning token findings are worth having. But if you are reading a launch post about a model that is hundreds of times cheaper than the alternative, the first question is not whether the multiple is real. It is whether the line item was ever big enough to care about. For most people running classification at ordinary volumes, it is not.

Latency: this is where the difference actually is#

Serial, one request at a time, from a laptop in Europe.

Jev is 2.6 times faster than the next fastest at the median, and the gap widens in the tail: 616 milliseconds at p95 against 1,670. The reasoning models are in another category, and gpt-oss-20b has a ten second p95.

This is the one dimension where the difference is large enough to survive how small our sample is, and it is structural rather than a tuning artefact. Jev does not generate anything. The others think first, and thinking takes time you pay for twice, once on the clock and once on the bill.

For a classifier sitting inline in a message pipeline, that tail is the number that decides it.

What we are doing#

We are not moving our carrier classifier onto Jev. It is at ceiling on the suite that protects it, it does an extraction in the same call that Jev cannot do at all, and it reads untrusted third party mail, which is the worst place to spend a novelty budget.

We are using it somewhere else. We have an autoresponder detector in an internal tool that is a hardcoded list of English and Slovene phrases, while our outreach goes to Austria, Germany and Hungary. A German out of office walks straight through it. That is one yes or no question, in any language, with nothing to extract, on our own data, on a path where latency matters. It is the shape this class of model is actually for.

What this cannot tell you#

More than we would like, and we would rather say it than have it pointed out.

No subset of our cases covers all four verdicts. The neutral held-out cases contain no UNRELATED at all; the adversarial ones contain no ANSWER and no UNCLEAR. So none of these numbers is a four class accuracy estimate.

The held-out mixture is our challenge suite, not a production distribution. It is deliberately weighted towards cases we thought would be hard, including six written specifically against weaknesses TypeSafe documents for their own model.

One author wrote and labelled every case. We had three models from three vendors, none of them in the benchmark, relabel everything blind. Agreement was 0.794, which is substantial, and one case remains genuinely contested: a Slovene reply pointing at a tracking portal, which we call UNCLEAR and all three labellers called something else. That may be our label being wrong, or it may be that our audit rubric abbreviated a rule our production prompt states explicitly. We have not resolved it, so we are telling you about it.

And Jev only ever ran in one prompt condition, because its API takes instructions and criteria and nothing else. We originally described that as "its production shape is the minimal condition", which sounded reasonable and was wrong: its response to prompt tuning is unobserved, not zero.

The scripts are in our repo. If you re-run them and get something different, we would like to hear it.