WhichJudge / decision
ns compares Jev with 4o-mini only
Dead heat: Jev 95.4%, Mini 94.6%, TF-IDF 96.0%, the 2026 model 93.0%. Nothing separates the four after correction. Classical ML is enough here.
Replaces: Spam vs ham classifier on inbound SMS
Gold: ucirvine/sms_spam
- UCI SMS Spam, n=500, seed=11
Labels: ham / spam
| Model | Accuracy | 95% CI | Latency (p50 unless stated) | Cost per 1M decisions |
|---|---|---|---|---|
| Jev jev-1.13.0 | 95.4% | [93.2% - 96.9%] | 716 ms | $15.16 |
| gpt-4o-mini-2024-07-18 | 94.6% | [92.3% - 96.3%] | 987 ms | $18.38 |
| gpt-5.4-mini-2026-03-17 | 93.0% | [90.4% - 94.9%] | 897 ms | unpriced (105.2 tok/call) |
| Laya (self-hosted, convaiinnovations/laya) | 92.6% | [90.0% - 94.6%] | 192 ms local cpu, no network | $0 API, compute not priced |
| gpt-6-luna (OpenAI Decisions API) | 94.8% | [92.5% - 96.4%] | 1305 ms, one client, region unset, includes network | $15.82 (list $0.10 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef (Cloudflare Workers AI) | 93.4% | [90.9% - 95.3%] | 893 ms, one client, region unset, includes network | $42.42 (list $0.24 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef-flash (Cloudflare Workers AI) | 90.4% | [87.5% - 92.7%] | 686 ms, one client, region unset, includes network | $15.91 (list $0.09 per 1M input tokens, 2026-10-09) |
| TF-IDF + LR (train n=5,032) | 96.0% | [93.9% - 97.4%] | 42 us mean per row, batched | $0 |
McNemar Jev vs Mini: ns, p=0.48 (11 Jev-only correct, 7 Mini-only correct). TF-IDF vs Jev: ns.
At >=0.9: 98.6% on 86.0% of messages, 3.2pt above ungated, with ECE 0.015, the best-calibrated row on the board. Automate the slice, queue the rest.
Auto slice: at Jev confidence ≥ 0.9, 98.6% accurate on 86.0% of traffic, which is +3.2pt against 95.4% ungated. That threshold is the strongest one still covering at least half the traffic, picked by rule from the measured gates.
Calibration: ECE 0.015 (max bin gap 0.271). ECE says whether the confidence can be read as a probability. It does not say whether gating helps: thresholding only needs the model to rank its own answers, so a badly calibrated task can still gate well. High ECE costs you the right to quote the number, not the right to threshold on it.
Frozen samples 3275e721484a, schema 1c2729400c20.
receipts/sms_spam.json here is the slim copy: per call,
the id, gold, prediction, confidence, latency and the SHA-256 of the request and response.
The full request and response bodies are in the repository at
results/receipts/sms_spam.json
(text replaced by its hash; scripts/rehydrate.py restores it).
git clone https://github.com/RadRebelSam/whichjudge.dev
python3 scripts/verify_run.py # recounts this table from the receipts, no API key
python3 scripts/run_eval.py # hits the APIs again with your own keysIndependent bench, not affiliated with TypeSafe AI. Public gold labels, n=500, not a procurement study. All 11 decisions · source and data · contact@radrebeldeveloper.com
build e6d1f683a4 · this row from run 1 of 1, 2026-09-21T06:23Z