> ## Documentation Index
> Fetch the complete documentation index at: https://fuguai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark

> Every battery, measured with every engine it has a receipt for: generated from the receipts, not written by hand.

Each battery ships its labelled rows, and a receipt for each engine it was measured with (`hunch test --model <engine> --receipt`). This page is those receipts side by side. A row is one engine answering one question on the battery's own rows; accuracy and AUROC are against the battery's gold. An engine's number here is on this battery's data: measure it on yours before choosing.

`act` bars are set for the spec's own engine; another engine's probabilities mean something else, so its acting numbers aren't comparable and are left out.

## `agent-commands`

Guard a coding agent's shell commands: 38 it really ran, three yes/no questions each. The quickstart; about \$0.001 to run.

| Engine                    | Question   | Rows with gold | Accuracy | AUROC | Calibration error | Battery's tests     |
| ------------------------- | ---------- | -------------- | -------- | ----- | ----------------- | ------------------- |
| `jev-1.13.0`              | `destroys` | 38             | 97.4%    | 0.995 | 0.119             | pass                |
| `deepseek:deepseek-flash` | `destroys` | 38             | 97.4%    | 0.984 | 0.045             | pass                |
| `ollama:qwen2.5:0.5b`     | `destroys` | 38             | 79.0%    | 0.557 | 0.215             | fail: min\_accuracy |

## `rag-answers`

Check a RAG answer against its sources: does it say anything the passages don't support? 900 real answers marked by annotators, one yes/no question; AUROC 0.941; about \$0.03 to run.

| Engine                                     | Question      | Rows with gold | Accuracy | AUROC | Calibration error | Battery's tests     |
| ------------------------------------------ | ------------- | -------------- | -------- | ----- | ----------------- | ------------------- |
| `jev-1.13.0`                               | `unsupported` | 900            | 83.8%    | 0.941 | 0.208             | pass                |
| `deepseek:deepseek-flash`                  | `unsupported` | 900            | 77.3%    | 0.907 | 0.201             | fail: min\_accuracy |
| `ollama:bespoke-minicheck#rag-unsupported` | `unsupported` | 900            | 67.4%    | 0.840 | 0.298             | fail: min\_accuracy |
| `ollama:qwen2.5:0.5b`                      | `unsupported` | 900            | 21.2%    | 0.434 | 0.609             | fail: min\_accuracy |

## `tickets`

Route 40 made-up support tickets: which team, is it urgent, how frustrated (a choice, a yes/no and a score). About \$0.001 to run.

| Engine                    | Question     | Rows with gold | Accuracy | AUROC | Calibration error | Battery's tests     |
| ------------------------- | ------------ | -------------- | -------- | ----- | ----------------- | ------------------- |
| `jev-1.13.0`              | `department` | 40             | 100.0%   | –     | 0.003             | pass                |
| `deepseek:deepseek-flash` | `department` | 40             | 100.0%   | –     | 0.002             | pass                |
| `ollama:qwen2.5:0.5b`     | `department` | 40             | 32.5%    | –     | 0.220             | fail: min\_accuracy |