WhichJudge / decision
Don't compares Jev with 4o-mini only
Mini 72.4% beats Jev 65.6% by 6.8pt (Holm p=0.002), the same loss as v1 on equal prompts. Jev catches only 47.6% of hate speech; Mini 64.0%, the 2026 model 38.0%. TF-IDF is worst at 53.4%. The neighbouring Civil Comments row, a different task on CC0 gold, ranks the models the other way.
Replaces: Hate vs not-hate classifier
Gold: cardiffnlp/tweet_eval
- tweet_eval hate, n=500, seed=11
Labels: not_hate / hate
| Model | Accuracy | 95% CI | Latency (p50 unless stated) | Cost per 1M decisions |
|---|---|---|---|---|
| Jev jev-1.13.0 | 65.6% | [61.3% - 69.6%] | 724 ms | $14.86 |
| gpt-4o-mini-2024-07-18 | 72.4% | [68.3% - 76.1%] | 963 ms | $18.2 |
| gpt-5.4-mini-2026-03-17 | 64.2% | [59.9% - 68.3%] | 849 ms | unpriced (104.7 tok/call) |
| Laya (self-hosted, convaiinnovations/laya) | 62.8% | [58.5% - 66.9%] | 194 ms local cpu, no network | $0 API, compute not priced |
| gpt-6-luna (OpenAI Decisions API) | 57.6% | [53.2% - 61.9%] | 885 ms, one client, region unset, includes network | $15.47 (list $0.10 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef (Cloudflare Workers AI) | 61.6% | [57.3% - 65.8%] | 826 ms, one client, region unset, includes network | $41.17 (list $0.24 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef-flash (Cloudflare Workers AI) | 60.8% | [56.5% - 65.0%] | 646 ms, one client, region unset, includes network | $15.44 (list $0.09 per 1M input tokens, 2026-10-09) |
| TF-IDF + LR (train n=8,960) | 53.4% | [49.0% - 57.7%] | 37 us mean per row, batched | $0 |
McNemar Jev vs Mini: mini, p=0.000187 (22 Jev-only correct, 56 Mini-only correct). TF-IDF vs Jev: jev.
Gating buys the most lift on the board and still leaves a bad row: 78.9% on 53.0% at >=0.9, 13.3pt above ungated, and Mini is better anyway. Keep a specialist.
Auto slice: at Jev confidence ≥ 0.9, 78.9% accurate on 53.0% of traffic, which is +13.3pt against 65.6% ungated. That threshold is the strongest one still covering at least half the traffic, picked by rule from the measured gates.
Calibration: ECE 0.196 (max bin gap 0.303). ECE says whether the confidence can be read as a probability. It does not say whether gating helps: thresholding only needs the model to rank its own answers, so a badly calibrated task can still gate well. High ECE costs you the right to quote the number, not the right to threshold on it.
Frozen samples 5494a7c66287, schema 8721de4565ba.
receipts/content_hate.json here is the slim copy: per call,
the id, gold, prediction, confidence, latency and the SHA-256 of the request and response.
The full request and response bodies are in the repository at
results/receipts/content_hate.json
(text replaced by its hash; scripts/rehydrate.py restores it).
git clone https://github.com/RadRebelSam/whichjudge.dev
python3 scripts/verify_run.py # recounts this table from the receipts, no API key
python3 scripts/run_eval.py # hits the APIs again with your own keysIndependent bench, not affiliated with TypeSafe AI. Public gold labels, n=500, not a procurement study. All 11 decisions · source and data · contact@radrebeldeveloper.com
build e6d1f683a4 · this row from run 1 of 1, 2026-09-21T06:23Z