> ## Documentation Index
> Fetch the complete documentation index at: https://fuguai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Test in CI

> Set thresholds in the spec, and fail the build when a change makes a judgment worse.

A unit test checks that a function returns what you expect. A judgment can't be checked that way, because some of its answers will always be wrong. What you can check is how often it is wrong, and whether its confidence can be trusted.

## Thresholds live in the spec

Add a `tests:` block, keyed by question:

```yaml prototype/examples/swe_agent/patch_eval.yml theme={null}
tests:
  passes_tests:
    min_auroc: 0.70
    max_calibration_error: 0.10
```

`hunch test` checks each one, prints `PASS` or `FAIL`, and exits 1 if any fails. A check marked `severity: warn` prints `WARN` instead and never fails the build: for something you want to see on every run without blocking on it.

Here is a judgment that reads a coding agent's patch and predicts whether it will pass the project's tests, from the [SWE-agent cookbook](/cookbooks/swe-patches), run against the real test results:

```bash theme={null}
hunch test prototype/examples/swe_agent/patch_eval.yml --max-cost 0
```

```text theme={null}
passes_tests (noul, 200 rows)
  gold: 200 rows (200 from source, 0 from review)
  PASS accuracy 77.8% [weighted to the population] (min 0%)
  FAIL calibration error 0.147 [weighted to the population] (max 0.1)
  PASS AUROC 0.831 (100 yes / 100 no; 0.5 = coin toss; unaffected by base rate) (min 0.7)
```

The model ranks patches well: a passing patch usually gets a higher probability than a failing one. Yet the spec fails. When the model says a patch will pass, it is more confident than it has reason to be, and a pipeline that skipped the tests on its confident "yes" would ship broken patches. Ranking alone would not have shown that.

The other tests are `min_accuracy`, `min_act_accuracy` (accuracy among the answers acted on without a person) and `order_stability` for choices; the [spec reference](/reference/spec#tests) describes each. A question with no tests still gets its numbers printed; nothing can fail.

Two more kinds of check live in the same spec. [Examples](/reference/spec#examples) pin rows whose answers must not change. [Metrics](/reference/spec#metrics) check a rule built from several answers, such as "stop the command if any question says yes", against gold, with limits like `max_missed`.

<Note>
  Thresholds compare the point estimate, not the lower end of its interval. With 30 reviewed rows the interval is about ±15 points, so set `min_accuracy` a little below what you measured.
</Note>

## What CI needs

`test` compares answers with gold, so CI needs both.

**Gold** is your answer key column and your `*.reviews.csv` files. Commit them.

**Answers** come from the store. `test` asks the engine for any it lacks, which in CI would mean paying on every pull request. Two things keep that small:

* Cache the store between runs. Answers are stored by their exact input, so a restored store is always safe to reuse, and only changed rows or questions are asked again.
* Always pass `--max-cost`. If the missing answers would cost more, `test` asks nothing and exits 1.

If anything may be asked, CI also needs an API key: `TYPESAFE_API_KEY` for Jev, or your engine's key ([Environment](/reference/environment)).

## A GitHub Actions workflow

```yaml .github/workflows/judgments.yml theme={null}
name: judgments
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - name: Restore the answer store
        uses: actions/cache@v4
        with:
          path: .hunch/store.sqlite
          key: hunch-store-${{ github.run_id }}
          restore-keys: hunch-store-
      - name: Install hunch
        run: uv tool install hunch-ai
      - name: Test
        env:
          TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
        run: hunch test evals/ --max-cost 0.50
```

On GitHub Actions, `test` also adds a table to the run's summary page: per question, accuracy, its range and any failed check, and a line for each metric and for the examples. The same numbers are in the [results file](/reference/cli#the-results-file) for any other CI.

The cache key changes every run and `restore-keys` picks the newest store, so answers asked in one run are there for the next. hunch uses the nearest `.hunch/store.sqlite` at or above the spec's folder, or creates one at the repository root. Set `HUNCH_STORE` to choose another path.

To see what a pull request does to the answers, add a step with `hunch diff evals/ --against git:origin/main --max-cost 0.50`. It reports and does not fail the build.

## Reading a failure

| Exit code | Meaning                                         |
| --------- | ----------------------------------------------- |
| 0         | Every check passed                              |
| 1         | A check failed, or `test` stopped before asking |
| 2         | A spec has a lint error                         |

Exit 1 has two causes, and the last line of the log tells them apart. Here is the [quickstart](/quickstart)'s guard tested with `--max-cost 0` on a fresh store, before any answers exist:

```text theme={null}
command_guard: would ask 114 answers in 38 requests (~$0.0010), above --max-cost $0.0; nothing asked
```

A script can tell them apart without reading the log. A finished `test` writes its results to `.hunch/target/`, as JSON with a `passed` field; a stopped one writes nothing. See [the results file](/reference/cli#the-results-file).