Reference
Jev benchmarks: third-party evidence and methods
Reviewed November 10, 2026.
Use the tables below to read what each source actually measured — model version, sample, metric, and stated limits — before borrowing a headline into production policy.
Research results need a version and a protocol
| Source | Model and scope | Reported result | Conditions and limits |
|---|---|---|---|
| Deußer et al., arXiv 2609.37647v2, revised October 29, 2026 | jev-1.13.0, 37 datasets, 346,009 unique answered requests | Main run cost $9.15; mean client-side latency 0.36 s | 32 concurrent connections, up to 1,100 requests/minute; latency includes network; not a 1.14 or host test |
| Sardine, November 2, 2026 | One fraud incident, 29 known fraudulent transactions; exact Jev version not specified | Jev flagged 27/29, rules 26/29, Sonnet 18/29; about 100 ms per Jev call | Input under 1,000 tokens; known-positive set cannot establish false-positive rate; single broad risk question |
| Encima, October 8, 2026 | First 240 cases plus 120 Dutch examples; exact Jev version not specified | 91.7% vs 88.3% on the first set; 108/120 vs 106/120 on the Dutch set | Comparator is the Decisions API; mixed tasks; complete version and infrastructure details not disclosed |
| Liquid AI, October 15, 2026 | Vendor comparison of d1-ultra, d1, and Jev 1.14 across 112 evaluations | Liquid claims improved accuracy and calibration | Vendor-reported comparison, not an independent rerun by this site |
What the arXiv study actually measured
The paper's v2 full text pins Jev to 1.13.0 with one frozen template per dataset and full evaluation splits. Templates were piloted on 20 training or validation examples before evaluation. Some validation splits substitute for unavailable test labels.
Classification, routing, moderation, grounding, and rubric scoring use different metrics — accuracy, F1 variants, AUPRC, and Spearman correlation where appropriate. Primary results use 95% bootstrap intervals. Calibration uses 15-bin ECE and Brier score.
Choice calibration was stronger than some Noul settings. In UNFAIR-ToS, micro-F1 rose from 0.499 at a 0.5 threshold to 0.748 with thresholds tuned on a separate 1,000-example sample — evidence for application-specific threshold validation, not a universal recommendation.
The study reports weaknesses in low-resource languages, noisy labels, legal judgments, rubric scoring, and sequential actions. It did not measure run-to-run variance. Reported main run cost is $9.15; $9.40 when pilot and additional analyses are included — different scopes, not conflicting prices.
Paper: arXiv v2 full text.
What the Encima latency and cost figures mean
Encima's report gives mean latency of 0.41 s for Jev and 0.52 s for the Decisions API, and roughly $0.0187 and $0.0230 per 1,000 similar requests. Local GLiNER reports 0.17 s with no per-request API fee, but hardware still has a cost.
These figures describe Encima's task mix. They do not show Jev is always faster, that local deployment is cost-free, or that this host reproduces the same latency.
Source: Encima test report.
Illustrative budget arithmetic, not a benchmark
For this host, pricing lists $0.175–$0.42 per million input tokens and free output under its metering. At an assumed 1,000 billed Jev input tokens per call and the $0.42/M rate: Pricing.
| Calls | Assumed input tokens | Calculated input cost |
|---|---|---|
| 100,000 | 100 million | $42 |
| 1,000,000 | 1 billion | $420 |
| 10,000,000 | 10 billion | $4,200 |
Formula: calls × billed input tokens per call ÷ 1,000,000 × rate. Arithmetic estimates, not receipts. For d1-ultra per-question billing, see Jev vs d1-ultra. Jev vs d1-ultra.
A protocol for your own future evaluation
This is a plan, not work completed by this site. Freeze a labeled dataset and templates before running a holdout set. Record model version, service channel, request shape, number of questions, image size, client region, concurrency, timeout, retry policy, and errors. Record returned usage, not an assumed tokenizer count.
Measure task-appropriate quality, calibration, review coverage, and end-to-end p50/p95 latency. Report sample size and uncertainty. Keep tuning data separate and distinguish API charges from hardware and engineering cost.
Design policy with the judgment-and-code method and the hosted API contract — not a benchmark headline alone.
Sources
- Deußer et al., arXiv
- Sardine case report
- Encima test report
- Liquid d1-ultra vendor evaluation
- Official Jev model versions and prices
This independent third-party host is not affiliated with TypeSafe AI. Published results are attributed to their original authors. No independent measurements by jevtypesafe.org are claimed.
Quellenangabe: