WhichJudge / decision
Neither compares Jev with 4o-mini only
Jev 80.5% against Mini 74.5% is a significant 6.0pt win (Holm p=0.036), and it does not matter: Jev misses 39.0% of injections, Mini 51.0%, the 2026 model 40.0%, all at 0.0% false positives. TF-IDF at 90.5% beats both APIs. A security gate is judged on what gets through, and every model here lets 39% or more through.
Replaces: LLM guard deciding whether untrusted input may reach an agent
Gold: deepset/prompt-injections
- deepset/prompt-injections, train + test, multilingual, n=200 (rows left after v1), n=200, seed=11
Labels: injection / legitimate
| Model | Accuracy | 95% CI | Latency (p50 unless stated) | Cost per 1M decisions |
|---|---|---|---|---|
| Jev jev-1.13.0 | 80.5% | [74.5% - 85.4%] | 797 ms | $18.09 |
| gpt-4o-mini-2024-07-18 | 74.5% | [68.0% - 80.0%] | 1107 ms | $28.91 |
| gpt-5.4-mini-2026-03-17 | 80.0% | [73.9% - 85.0%] | 1058 ms | unpriced (175.0 tok/call) |
| Laya (self-hosted, convaiinnovations/laya) | 63.5% | [56.6% - 69.9%] | 356 ms local cpu, no network | $0 API, compute not priced |
| gpt-6-luna (OpenAI Decisions API) | 72.0% | [65.4% - 77.8%] | 814 ms, one client, region unset, includes network | $22.72 (list $0.10 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef (Cloudflare Workers AI) | 79.0% | [72.8% - 84.1%] | 966 ms, one client, region unset, includes network | $58.69 (list $0.24 per 1M input tokens, 2026-10-09) |
| @cf/cloudflare/clef-flash (Cloudflare Workers AI) | 68.0% | [61.2% - 74.1%] | 648 ms, one client, region unset, includes network | $22.01 (list $0.09 per 1M input tokens, 2026-10-09) |
| TF-IDF + LR (train n=461) | 90.5% | [85.6% - 93.8%] | 39 us mean per row, batched | $0 |
McNemar Jev vs Mini: jev, p=0.00596 (14 Jev-only correct, 2 Mini-only correct). TF-IDF vs Jev: tfidf.
At >=0.9: 86.3% on 84.0% of rows, 5.8pt above ungated. Recall of injections on the covered rows barely moves, so raising the bar hands the misses to a human instead of catching them.
Auto slice: at Jev confidence ≥ 0.9, 86.3% accurate on 84.0% of traffic, which is +5.8pt against 80.5% ungated. That threshold is the strongest one still covering at least half the traffic, picked by rule from the measured gates.
Calibration: ECE 0.137 (max bin gap 0.47). ECE says whether the confidence can be read as a probability. It does not say whether gating helps: thresholding only needs the model to rank its own answers, so a badly calibrated task can still gate well. High ECE costs you the right to quote the number, not the right to threshold on it.
Frozen samples 7288f7ddd121, schema 6c6520e72c9f.
receipts/prompt_injection.json here is the slim copy: per call,
the id, gold, prediction, confidence, latency and the SHA-256 of the request and response.
The full request and response bodies are in the repository at
results/receipts/prompt_injection.json
.
git clone https://github.com/RadRebelSam/whichjudge.dev
python3 scripts/verify_run.py # recounts this table from the receipts, no API key
python3 scripts/run_eval.py # hits the APIs again with your own keysIndependent bench, not affiliated with TypeSafe AI. Public gold labels, n=200, not a procurement study. All 11 decisions · source and data · contact@radrebeldeveloper.com
build e6d1f683a4 · this row from run 1 of 1, 2026-09-21T06:23Z