What has TypeSafe published?
TypeSafe reports headline gains of 193.6× faster and 444.6× cheaper from its workflow evaluations. Its launch article qualifies those figures as potentially near the high end of practical gains. We have not reproduced them.
Read the official workflow evaluation site alongside the claim. It covers security incidents, agent-trace review, invoice processing, and customer service. The overview averages four workflows equally and uses model-generated consensus labels as its reference.
The results are evidence about a particular evaluation design. They do not establish the same speedup, savings, or correctness on every user’s data.
Read the reference answer before the chart
TypeSafe’s methodology overview describes a fixed workflow with programmatic rules and narrow model judgments. Its reference labels combine outputs from GPT-6 Astra and Claude Fable 5.1 at high thinking; evaluated models use provider-default reasoning settings.
Our interpretation: matching that reference is a useful comparison inside the published experiment, but it is different from matching independently verified answers from your own domain. A label generated by another model can still be mistaken. Inspect disagreements and the workflow rules before translating chart positions into a deployment decision.
The published adapter can help compare LLM-backed decisions using a similar interface. Document its settings too; retries and probability-output requirements affect the work performed.
Design an evaluation for your actual task
The following is our suggested evaluation method, not a claim about measured Jev performance:
- Choose one outcome. For an inbox, use correct team assignment rather than a vague “understands the message” score.
- Prepare representative cases. Include ordinary messages, mixed topics, missing context, irrelevant text, and examples in the languages you serve.
- Label before running. Record the expected answer and mark genuinely ambiguous cases for review.
- Separate tuning from testing. Refine criteria on one set and assess the final settings on another.
- Repeat under realistic conditions. Use the deployment region and expected concurrency; keep failed requests in the results.
Do not choose a dataset size because it makes a chart look decisive. Start small enough to inspect every error, then expand to cover the cases and error rate that matter to your workflow.
Record quality, time, and total work
| Measure | Record | Why it matters |
|---|---|---|
| Decision quality | Correct, incorrect, and unresolved cases | A forced answer can conceal uncertainty |
| Review rate | Share of cases sent to a human | Automation coverage changes operating cost |
| Latency | Median, slow-case percentiles, and errors | An average can hide slow requests |
| Spend | Usage, billed cost, and retries | Nominal token price is only one input |
| Reproducibility | Model, SDK, date, region, and criteria | A later comparison needs the same conditions |
For a routing example, report how many messages were assigned correctly without review and inspect where the wrong assignments went. Compare against a simple rule-based baseline as well as a model alternative.
What the benchmarks cannot settle for you
A workload benchmark does not by itself validate your access controls, billing logic, data policy, or fallback procedure. Those are parts of the surrounding application.
Read TypeSafe’s Jev 1.13 limitations before choosing examples. Our limitations guide and Jev versus LLM comparison help turn the results into a narrower decision: whether this model meets the requirements of one particular workflow.
Sources checked September 23, 2026. This is an independent guide; access, packages, and provider terms can change.
Sources & verification
- TypeSafe launch: results and qualifications ↗
- TypeSafe workflow evaluations ↗
- Published model comparison adapter ↗
- Version-specific limitations ↗
Checked September 23, 2026. This guide summarizes documentation; it is not an independent benchmark. Provider details can change.