WhichJudge / decision
ns compares Jev with 4o-mini only
Tied: Jev 96.3% vs Mini 95.5% (Holm p=1). TF-IDF drops to 84.8%, so the models earn their keep against classical, not against each other. n=356 because v1 used the rest of the split.
Replaces: Binary review sentiment
Gold: stanfordnlp/sst2
- SST-2 validation, the 356 rows left after v1, n=356, seed=11
Labels: negative / positive
| Model | Accuracy | 95% CI | Latency (p50 unless stated) | Cost per 1M decisions |
|---|---|---|---|---|
| Jev jev-1.13.0 | 96.3% | [93.9% - 97.9%] | 720 ms | $14.06 |
| gpt-4o-mini-2024-07-18 | 95.5% | [92.8% - 97.2%] | 1007 ms | $14.96 |
| gpt-5.4-mini-2026-03-17 | 97.2% | [94.9% - 98.5%] | 857 ms | unpriced (84.6 tok/call) |
| Laya (self-hosted, convaiinnovations/laya) | 91.6% | [88.2% - 94.0%] | 160 ms local cpu, no network | $0 API, compute not priced |
| gpt-6-luna (OpenAI Decisions API) | 93.8% | [90.8% - 95.9%] | 1339 ms, one client, region unset, includes network | $13.82 (list $0.10 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef (Cloudflare Workers AI) | 96.9% | [94.6% - 98.3%] | 941 ms, one client, region unset, includes network | $36.68 (list $0.24 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef-flash (Cloudflare Workers AI) | 96.6% | [94.2% - 98.1%] | 664 ms, one client, region unset, includes network | $13.76 (list $0.09 per 1M input tokens, 2026-10-09) |
| TF-IDF + LR (train n=67,349) | 84.8% | [80.7% - 88.2%] | 37 us mean per row, batched | $0 |
McNemar Jev vs Mini: ns, p=0.546 (7 Jev-only correct, 4 Mini-only correct). TF-IDF vs Jev: jev.
At >=0.8: 97.9% on 93.3% of reviews, 1.5pt above ungated. There is no headroom left, and ECE 0.030 says the confidence may be quoted.
Auto slice: at Jev confidence ≥ 0.8, 97.9% accurate on 93.3% of traffic, which is +1.5pt against 96.3% ungated. That threshold is the strongest one still covering at least half the traffic, picked by rule from the measured gates.
Calibration: ECE 0.03 (max bin gap 0.217). ECE says whether the confidence can be read as a probability. It does not say whether gating helps: thresholding only needs the model to rank its own answers, so a badly calibrated task can still gate well. High ECE costs you the right to quote the number, not the right to threshold on it.
Frozen samples 05ae16616476, schema f3bdb40526a0.
receipts/review_sentiment.json here is the slim copy: per call,
the id, gold, prediction, confidence, latency and the SHA-256 of the request and response.
The full request and response bodies are in the repository at
results/receipts/review_sentiment.json
(text replaced by its hash; scripts/rehydrate.py restores it).
git clone https://github.com/RadRebelSam/whichjudge.dev
python3 scripts/verify_run.py # recounts this table from the receipts, no API key
python3 scripts/run_eval.py # hits the APIs again with your own keysIndependent bench, not affiliated with TypeSafe AI. Public gold labels, n=356, not a procurement study. All 11 decisions · source and data · contact@radrebeldeveloper.com
build e6d1f683a4 · this row from run 1 of 1, 2026-09-21T06:23Z