WhichJudge / decision
Replace compares Jev with 4o-mini only
Jev 89.8% beats Mini 86.0% by 3.8pt (Holm p=0.025), and the 2026 model at 86.8% ties Jev. Laya, self-hosted, scores 93.4%, ns against Jev after correction; TF-IDF at 93.8% after training on 120k rows beats both APIs. If you have labels, skip both.
Replaces: 4-way news classifier
Gold: fancyzhx/ag_news
- AG News test, n=500, seed=11
Labels: world / sports / business / sci_tech
| Model | Accuracy | 95% CI | Latency (p50 unless stated) | Cost per 1M decisions |
|---|---|---|---|---|
| Jev jev-1.13.0 | 89.8% | [86.8% - 92.2%] | 719 ms | $16.96 |
| gpt-4o-mini-2024-07-18 | 86.0% | [82.7% - 88.8%] | 987 ms | $24.21 |
| gpt-5.4-mini-2026-03-17 | 86.8% | [83.6% - 89.5%] | 827 ms | unpriced (143.6 tok/call) |
| Laya (self-hosted, convaiinnovations/laya) | 93.4% | [90.9% - 95.3%] | 254 ms local cpu, no network | $0 API, compute not priced |
| gpt-6-luna (OpenAI Decisions API) | 88.4% | [85.3% - 90.9%] | 1161 ms, one client, region unset, includes network | $19.31 (list $0.10 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef (Cloudflare Workers AI) | 91.2% | [88.4% - 93.4%] | 1233 ms, one client, region unset, includes network | $54.82 (list $0.24 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef-flash (Cloudflare Workers AI) | 91.2% | [88.4% - 93.4%] | 699 ms, one client, region unset, includes network | $20.56 (list $0.09 per 1M input tokens, 2026-10-09) |
| TF-IDF + LR (train n=120,000) | 93.8% | [91.3% - 95.6%] | 60 us mean per row, batched | $0 |
McNemar Jev vs Mini: jev, p=0.00308 (28 Jev-only correct, 9 Mini-only correct). TF-IDF vs Jev: tfidf.
Among APIs Jev 89.8% ties clef and clef-flash, both 91.2%, after correction. At >=0.9: 94.0% on 87.4% of articles, 4.2pt above ungated, ECE 0.068.
Auto slice: at Jev confidence ≥ 0.9, 94.0% accurate on 87.4% of traffic, which is +4.2pt against 89.8% ungated. That threshold is the strongest one still covering at least half the traffic, picked by rule from the measured gates.
Calibration: ECE 0.068 (max bin gap 0.24). ECE says whether the confidence can be read as a probability. It does not say whether gating helps: thresholding only needs the model to rank its own answers, so a badly calibrated task can still gate well. High ECE costs you the right to quote the number, not the right to threshold on it.
Frozen samples 01f2ea5de94a, schema 33f5f9f90e44.
receipts/news_topic.json here is the slim copy: per call,
the id, gold, prediction, confidence, latency and the SHA-256 of the request and response.
The full request and response bodies are in the repository at
results/receipts/news_topic.json
(text replaced by its hash; scripts/rehydrate.py restores it).
git clone https://github.com/RadRebelSam/whichjudge.dev
python3 scripts/verify_run.py # recounts this table from the receipts, no API key
python3 scripts/run_eval.py # hits the APIs again with your own keysIndependent bench, not affiliated with TypeSafe AI. Public gold labels, n=500, not a procurement study. All 11 decisions · source and data · contact@radrebeldeveloper.com
build e6d1f683a4 · this row from run 1 of 1, 2026-09-21T06:23Z