WhichJudge / decision
Replace compares Jev with 4o-mini only
A neighbour of the hate-speech row: toxicity in comments rather than hate speech aimed at a group in tweets, on CC0 gold. Jev 76.8% beats Mini 72.4% (Holm p=0.035). TF-IDF beats both at 85.2%, and clef, a hosted typed-decision API, beats Jev at 86.0%, as clef-flash beats Jev at 85.4%. Read this next to the hate row: a different task and dataset gives a different ranking, which is the point, not proof that the hate loss was label noise.
Replaces: Toxicity classifier gating user comments before publication
Gold: google/civil_comments
- Civil Comments test split (CC0), n=500, seed=11
Labels: toxic / not_toxic
| Model | Accuracy | 95% CI | Latency (p50 unless stated) | Cost per 1M decisions |
|---|---|---|---|---|
| Jev jev-1.13.0 | 76.8% | [72.9% - 80.3%] | 770 ms | $19.27 |
| gpt-4o-mini-2024-07-18 | 72.4% | [68.3% - 76.1%] | 996 ms | $33.87 |
| gpt-5.4-mini-2026-03-17 | 75.6% | [71.6% - 79.2%] | 892 ms | unpriced (203.6 tok/call) |
| Laya (self-hosted, convaiinnovations/laya) | 72.2% | [68.1% - 75.9%] | 391 ms local cpu, no network | $0 API, compute not priced |
| gpt-6-luna (OpenAI Decisions API) | 72.6% | [68.5% - 76.3%] | 903 ms, one client, region unset, includes network | $25.33 (list $0.10 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef (Cloudflare Workers AI) | 86.0% | [82.7% - 88.8%] | 1167 ms, one client, region unset, includes network | $64.68 (list $0.24 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef-flash (Cloudflare Workers AI) | 85.4% | [82.0% - 88.2%] | 1286 ms, one client, region unset, includes network | $24.25 (list $0.09 per 1M input tokens, 2026-10-09) |
| TF-IDF + LR (train n=60,000) | 85.2% | [81.8% - 88.0%] | 77 us mean per row, batched | $0 |
McNemar Jev vs Mini: jev, p=0.00501 (39 Jev-only correct, 17 Mini-only correct). TF-IDF vs Jev: tfidf.
86.7% on 58.8% of comments at >=0.9, 9.9pt above ungated, and ECE 0.101 is low enough to state the confidence out loud. Publish that slice automatically and queue the rest.
Auto slice: at Jev confidence ≥ 0.9, 86.7% accurate on 58.8% of traffic, which is +9.9pt against 76.8% ungated. That threshold is the strongest one still covering at least half the traffic, picked by rule from the measured gates.
Calibration: ECE 0.101 (max bin gap 0.13). ECE says whether the confidence can be read as a probability. It does not say whether gating helps: thresholding only needs the model to rank its own answers, so a badly calibrated task can still gate well. High ECE costs you the right to quote the number, not the right to threshold on it.
Frozen samples 52e19e1682d7, schema 99a600702dd1.
receipts/civil_toxicity.json here is the slim copy: per call,
the id, gold, prediction, confidence, latency and the SHA-256 of the request and response.
The full request and response bodies are in the repository at
results/receipts/civil_toxicity.json
.
git clone https://github.com/RadRebelSam/whichjudge.dev
python3 scripts/verify_run.py # recounts this table from the receipts, no API key
python3 scripts/run_eval.py # hits the APIs again with your own keysIndependent bench, not affiliated with TypeSafe AI. Public gold labels, n=500, not a procurement study. All 11 decisions · source and data · contact@radrebeldeveloper.com
build e6d1f683a4 · this row from run 1 of 1, 2026-09-21T06:23Z