WhichJudge / decision
Replace compares Jev with 4o-mini only
TF-IDF 94.2% beats both neural judges by more than 20 points if you have labels. Among APIs Jev 70.2% beats Mini 66.4% (Holm p=0.021) and ties the 2026 model at 72.4%; clef beats Jev at 76.2% and clef-flash beats Jev at 81.4%, and both still trail TF-IDF. The 'other' queue is the failure: Jev routes 1.6% of it correctly. Do not pay for a model on this task if you can train one.
Replaces: Support-queue classifier on banking tickets
Gold: PolyAI BANKING77
- BANKING77 train split, collapsed 77 to 8 queues, n=500, seed=11
Labels: cards / top_up / cash_atm / transfers / fx / account / payments_fees / other
| Model | Accuracy | 95% CI | Latency (p50 unless stated) | Cost per 1M decisions |
|---|---|---|---|---|
| Jev jev-1.13.0 | 70.2% | [66.0% - 74.0%] | 694 ms | $20.65 |
| gpt-4o-mini-2024-07-18 | 66.4% | [62.1% - 70.4%] | 1054 ms | $34.81 |
| gpt-5.4-mini-2026-03-17 | 72.4% | [68.3% - 76.1%] | 847 ms | unpriced (216.8 tok/call) |
| Laya (self-hosted, convaiinnovations/laya) | 56.8% | [52.4% - 61.1%] | 370 ms local cpu, no network | $0 API, compute not priced |
| gpt-6-luna (OpenAI Decisions API) | 70.2% | [66.0% - 74.0%] | 1316 ms, one client, region unset, includes network | $25.74 (list $0.10 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef (Cloudflare Workers AI) | 76.2% | [72.3% - 79.7%] | 1152 ms, one client, region unset, includes network | $80.33 (list $0.24 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef-flash (Cloudflare Workers AI) | 81.4% | [77.8% - 84.6%] | 715 ms, one client, region unset, includes network | $30.12 (list $0.09 per 1M input tokens, 2026-10-09) |
| TF-IDF + LR (train n=9,503) | 94.2% | [91.8% - 95.9%] | 18 us mean per row, batched | $0 |
McNemar Jev vs Mini: jev, p=0.00235 (27 Jev-only correct, 8 Mini-only correct). TF-IDF vs Jev: tfidf.
Gating works here even though the calibration does not: at >=0.9 it is 79.0% on 80.2% of tickets, 8.9pt above ungated. What ECE 0.232 costs you is the right to quote the confidence as a probability, not the right to threshold on it.
Auto slice: at Jev confidence ≥ 0.9, 79.0% accurate on 80.2% of traffic, which is +8.8pt against 70.2% ungated. That threshold is the strongest one still covering at least half the traffic, picked by rule from the measured gates.
Calibration: ECE 0.232 (max bin gap 0.59). ECE says whether the confidence can be read as a probability. It does not say whether gating helps: thresholding only needs the model to rank its own answers, so a badly calibrated task can still gate well. High ECE costs you the right to quote the number, not the right to threshold on it.
Frozen samples 7510a530521d, schema c898ec217cfd.
receipts/banking_coarse_route.json here is the slim copy: per call,
the id, gold, prediction, confidence, latency and the SHA-256 of the request and response.
The full request and response bodies are in the repository at
results/receipts/banking_coarse_route.json
.
git clone https://github.com/RadRebelSam/whichjudge.dev
python3 scripts/verify_run.py # recounts this table from the receipts, no API key
python3 scripts/run_eval.py # hits the APIs again with your own keysIndependent bench, not affiliated with TypeSafe AI. Public gold labels, n=500, not a procurement study. All 11 decisions · source and data · contact@radrebeldeveloper.com
build e6d1f683a4 · this row from run 1 of 1, 2026-09-21T06:23Z