Benchmark

Every battery, measured with every engine it has a receipt for: generated from the receipts, not written by hand.

Each battery ships its labelled rows, and a receipt for each engine it was measured with (hunch test --model <engine> --receipt). This page is those receipts side by side. A row is one engine answering one question on the battery’s own rows; accuracy and AUROC are against the battery’s gold. An engine’s number here is on this battery’s data: measure it on yours before choosing.

act bars are set for the spec’s own engine; another engine’s probabilities mean something else, so its acting numbers aren’t comparable and are left out.

agent-commands

Guard a coding agent’s shell commands: 38 it really ran, three yes/no questions each. The quickstart; about $0.001 to run.

Engine	Question	Rows with gold	Accuracy	AUROC	Calibration error	Battery’s tests
jev-1.13.0	destroys	38	97.4%	1.000	0.099	pass
deepseek:deepseek-flash	destroys	38	97.4%	1.000	0.054	pass
ollama:qwen2.5:0.5b	destroys	38	71.0%	0.547	0.239	fail: min_accuracy

rag-answers

Check a RAG answer against its sources: does it say anything the passages don’t support? 900 real answers marked by annotators, one yes/no question; AUROC 0.941; about $0.03 to run.

Engine	Question	Rows with gold	Accuracy	AUROC	Calibration error	Battery’s tests
jev-1.13.0	unsupported	900	83.8%	0.941	0.208	pass
deepseek:deepseek-flash	unsupported	900	77.3%	0.907	0.201	fail: min_accuracy
ollama:bespoke-minicheck#rag-unsupported	unsupported	900	67.4%	0.840	0.298	fail: min_accuracy
ollama:qwen2.5:0.5b	unsupported	900	21.2%	0.434	0.609	fail: min_accuracy

tickets

Route 40 made-up support tickets: which team, is it urgent, how frustrated (a choice, a yes/no and a score). About $0.001 to run.

Engine	Question	Rows with gold	Accuracy	AUROC	Calibration error	Battery’s tests
jev-1.13.0	department	40	100.0%	–	0.003	pass
deepseek:deepseek-flash	department	40	100.0%	–	0.002	pass
ollama:qwen2.5:0.5b	department	40	32.5%	–	0.220	fail: min_accuracy