prototype/examples/.
Several cases use AI review panels, source labels or the author’s own reviews. Those are different forms of evidence, and none alone establishes how a spec will work on your data. Each case’s measurement notes say which it used.
Grade Claude Code sessions
Did each turn work? About 90% right against reviewers, and
diff caught a wording change that made it worse.Guard a coding agent's commands
Stops 11.6% of 1,315 real commands and lets through an estimated 1% (0.2–6%) that needed a person.
Check RAG answers against their sources
Passes 54% of 900 annotated answers as supported with 4 wrong, and ranks six models as the annotators do.
Read an NDA before a lawyer does
One word in the criteria fixed 21 of 61 contracts; still a first pass, not a decision.
Match two product catalogues
Rules in SQL, one question per pair, SQL again for the matches: 971 of 980 matches right, and the 682 it is sure of all right.
Find what a lawsuit asks for
455,449 Enron emails. The request’s words find 8 in 100 of what the lawyer wanted; the lawyer’s reading, written into the spec, finds 63.
Screen a literature search in the right order
Three real systematic reviews, their written criteria as the question. Against active learning, the field’s own tool: it wins one, hunch wins one, and on the third they find different papers first.
Catch comments a change left wrong
A parser finds what changed, one question judges each comment: 85.7% on 300 hand-checked comments, and a check that fails CI on sure ones.
Route banking intents
77 intents. The answer key turned out to be wrong more often than the model.
Predict whether a patch passes
A confident “no” is reliable; a confident “yes” is not.
100,000 product reviews
100,000 rows: 96,000 new requests at 57.5 per second for $1.55, then a cached re-run in 4 seconds.