WhichJudge / decision
ns compares Jev with 4o-mini only
Tied after correction: Jev 78.2%, Mini 76.2%, the 2026 model 76.2% (Holm p=1). v1 had this as a Replace on the label-names-only prompt; with the same rubric on both sides the gap is 2.0pt and not significant. Optimism is still the weak class: Jev catches 62.4% of it, Mini 52.0%. TF-IDF trails at 60.0%.
Replaces: 4-way emotion classifier on short messages
Gold: cardiffnlp/tweet_eval
- tweet_eval emotion, train split (test exhausted by v1), n=500, seed=11
Labels: anger / joy / optimism / sadness
| Model | Accuracy | 95% CI | Latency (p50 unless stated) | Cost per 1M decisions |
|---|---|---|---|---|
| Jev jev-1.13.0 | 78.2% | [74.4% - 81.6%] | 733 ms | $15.44 |
| gpt-4o-mini-2024-07-18 | 76.2% | [72.3% - 79.7%] | 968 ms | $18.29 |
| gpt-5.4-mini-2026-03-17 | 76.2% | [72.3% - 79.7%] | 876 ms | unpriced (104.2 tok/call) |
| Laya (self-hosted, convaiinnovations/laya) | 69.4% | [65.2% - 73.3%] | 190 ms local cpu, no network | $0 API, compute not priced |
| gpt-6-luna (OpenAI Decisions API) | 75.6% | [71.6% - 79.2%] | 855 ms, one client, region unset, includes network | $15.48 (list $0.10 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef (Cloudflare Workers AI) | 78.4% | [74.6% - 81.8%] | 770 ms, one client, region unset, includes network | $46.34 (list $0.24 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef-flash (Cloudflare Workers AI) | 76.4% | [72.5% - 79.9%] | 667 ms, one client, region unset, includes network | $17.38 (list $0.09 per 1M input tokens, 2026-10-09) |
| TF-IDF + LR (train n=2,757) | 60.0% | [55.6% - 64.2%] | 24 us mean per row, batched | $0 |
McNemar Jev vs Mini: ns, p=0.229 (33 Jev-only correct, 23 Mini-only correct). TF-IDF vs Jev: jev.
At >=0.9: 88.8% on 64.2% of messages, 10.6pt above ungated. ECE 0.100: threshold on it, do not quote it.
Auto slice: at Jev confidence ≥ 0.9, 88.8% accurate on 64.2% of traffic, which is +10.6pt against 78.2% ungated. That threshold is the strongest one still covering at least half the traffic, picked by rule from the measured gates.
Calibration: ECE 0.1 (max bin gap 0.173). ECE says whether the confidence can be read as a probability. It does not say whether gating helps: thresholding only needs the model to rank its own answers, so a badly calibrated task can still gate well. High ECE costs you the right to quote the number, not the right to threshold on it.
Frozen samples 10568e3b3a3f, schema c02bb1f03f4e.
receipts/message_emotion.json here is the slim copy: per call,
the id, gold, prediction, confidence, latency and the SHA-256 of the request and response.
The full request and response bodies are in the repository at
results/receipts/message_emotion.json
(text replaced by its hash; scripts/rehydrate.py restores it).
git clone https://github.com/RadRebelSam/whichjudge.dev
python3 scripts/verify_run.py # recounts this table from the receipts, no API key
python3 scripts/run_eval.py # hits the APIs again with your own keysIndependent bench, not affiliated with TypeSafe AI. Public gold labels, n=500, not a procurement study. All 11 decisions · source and data · contact@radrebeldeveloper.com
build e6d1f683a4 · this row from run 1 of 1, 2026-09-21T06:23Z