WhichJudge / decision
Replace compares Jev with 4o-mini only
The only row here that is not academic gold: a regulator runs this routing live and the consumer writes the narrative. Jev 83.8% beats Mini 77.4% by 6.4pt (Holm p=0.0002). TF-IDF scores 85.6%, ns against Jev and significant against Mini; the 2026 model ties Jev at 83.8%. Narratives average about 960 characters, past the cost crossover, so Jev is the cheaper API by the widest margin on the board: $33.42 against $78.59 per million decisions.
Replaces: LLM classifier routing inbound financial complaints to a product queue
Gold: CFPB Consumer Complaint Database
via a CC0 mirror
- CFPB Consumer Complaint Database, 8 product queues, n=500, seed=11
Labels: credit_reporting / debt_collection / cards / bank_account / mortgage / money_transfer / loans / student_loan
| Model | Accuracy | 95% CI | Latency (p50 unless stated) | Cost per 1M decisions |
|---|---|---|---|---|
| Jev jev-1.13.0 | 83.8% | [80.3% - 86.8%] | 712 ms | $33.42 |
| gpt-4o-mini-2024-07-18 | 77.4% | [73.5% - 80.8%] | 960 ms | $78.59 |
| gpt-5.4-mini-2026-03-17 | 83.8% | [80.3% - 86.8%] | 822 ms | unpriced (506.0 tok/call) |
| Laya (self-hosted, convaiinnovations/laya) | 57.0% | [52.6% - 61.3%] | 970 ms local cpu, no network | $0 API, compute not priced |
| gpt-6-luna (OpenAI Decisions API) | 83.0% | [79.5% - 86.0%] | 802 ms, one client, region unset, includes network | $54.61 (list $0.10 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef (Cloudflare Workers AI) | 82.2% | [78.6% - 85.3%] | 1429 ms, one client, region unset, includes network | $153.76 (list $0.24 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef-flash (Cloudflare Workers AI) | 77.0% | [73.1% - 80.5%] | 621 ms, one client, region unset, includes network | $57.66 (list $0.09 per 1M input tokens, 2026-10-09) |
| TF-IDF + LR (train n=63,789) | 85.6% | [82.3% - 88.4%] | 288 us mean per row, batched | $0 |
McNemar Jev vs Mini: jev, p=1.72e-05 (42 Jev-only correct, 10 Mini-only correct). TF-IDF vs Jev: ns.
At >=0.9: 92.1% on 78.6% of complaints, 8.3pt above ungated. ECE 0.094 means do not repeat the confidence to anyone as a probability; it does not stop you thresholding on it.
Auto slice: at Jev confidence ≥ 0.9, 92.1% accurate on 78.6% of traffic, which is +8.3pt against 83.8% ungated. That threshold is the strongest one still covering at least half the traffic, picked by rule from the measured gates.
Calibration: ECE 0.094 (max bin gap 0.298). ECE says whether the confidence can be read as a probability. It does not say whether gating helps: thresholding only needs the model to rank its own answers, so a badly calibrated task can still gate well. High ECE costs you the right to quote the number, not the right to threshold on it.
Frozen samples d6d9ae005e10, schema cd991970d17b.
receipts/cfpb_queue_route.json here is the slim copy: per call,
the id, gold, prediction, confidence, latency and the SHA-256 of the request and response.
The full request and response bodies are in the repository at
results/receipts/cfpb_queue_route.json
.
git clone https://github.com/RadRebelSam/whichjudge.dev
python3 scripts/verify_run.py # recounts this table from the receipts, no API key
python3 scripts/run_eval.py # hits the APIs again with your own keysIndependent bench, not affiliated with TypeSafe AI. Public gold labels, n=500, not a procurement study. All 11 decisions · source and data · contact@radrebeldeveloper.com
build e6d1f683a4 · this row from run 1 of 1, 2026-09-21T06:23Z