Test in CI

Set thresholds in the spec, and fail the build when a change makes a judgment worse.

A unit test checks that a function returns what you expect. A judgment can’t be checked that way, because some of its answers will always be wrong. What you can check is how often it is wrong, and whether its confidence can be trusted.

Thresholds live in the spec

Add a tests: block, keyed by question:

prototype/examples/swe_agent/patch_eval.yml

tests:
  passes_tests:
    min_auroc: 0.70
    max_calibration_error: 0.10

hunch test checks each one, prints PASS or FAIL, and exits 1 if any fails. A check marked severity: warn prints WARN instead and never fails the build: for something you want to see on every run without blocking on it.

Here is a judgment that reads a coding agent’s patch and predicts whether it will pass the project’s tests, from the SWE-agent cookbook, run against the real test results:

hunch test prototype/examples/swe_agent/patch_eval.yml --max-cost 0

passes_tests (noul, 200 rows)
  gold: 200 rows (200 from source, 0 from review)
  PASS accuracy 77.8% [weighted to the population] (min 0%)
  FAIL calibration error 0.147 [weighted to the population] (max 0.1)
  PASS AUROC 0.831 (100 yes / 100 no; 0.5 = coin toss; unaffected by base rate) (min 0.7)

The model ranks patches well: a passing patch usually gets a higher probability than a failing one. Yet the spec fails. When the model says a patch will pass, it is more confident than it has reason to be, and a pipeline that skipped the tests on its confident “yes” would ship broken patches. Ranking alone would not have shown that.

The other tests are min_accuracy, min_act_accuracy (accuracy among the answers acted on without a person) and order_stability for choices; the spec reference describes each. A question with no tests still gets its numbers printed; nothing can fail.

Two more kinds of check live in the same spec. Examples pin rows whose answers must not change. Metrics check a rule built from several answers, such as “stop the command if any question says yes”, against gold, with limits like max_missed.

Thresholds compare the point estimate, not the lower end of its interval. With 30 reviewed rows the interval is about ±15 points, so set min_accuracy a little below what you measured.

What CI needs

test compares answers with gold, so CI needs both.

Gold is your answer key column and your *.reviews.csv files. Commit them.

Answers come from the store. test asks the engine for any it lacks, which in CI would mean paying on every pull request. Two things keep that small:

• Cache the store between runs. Answers are stored by their exact input, so a restored store is always safe to reuse, and only changed rows or questions are asked again.
• Always pass --max-cost. If the missing answers would cost more, test asks nothing and exits 1.

If anything may be asked, CI also needs an API key: TYPESAFE_API_KEY for Jev, or your engine’s key (Environment).

A GitHub Actions workflow

.github/workflows/judgments.yml

name: judgments
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - name: Restore the answer store
        uses: actions/cache@v4
        with:
          path: .hunch/store.sqlite
          key: hunch-store-${{ github.run_id }}
          restore-keys: hunch-store-
      - name: Install hunch
        run: uv tool install hunch-ai
      - name: Test
        env:
          TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
        run: hunch test evals/ --max-cost 0.50

On GitHub Actions, test also adds a table to the run’s summary page: per question, accuracy, its range and any failed check, and a line for each metric and for the examples. The same numbers are in the results file for any other CI.

The cache key changes every run and restore-keys picks the newest store, so answers asked in one run are there for the next. hunch uses the nearest .hunch/store.sqlite at or above the spec’s folder, or creates one at the repository root. Set HUNCH_STORE to choose another path.

To see what a pull request does to the answers, add a step with hunch diff evals/ --against git:origin/main --max-cost 0.50. It reports and does not fail the build.

Reading a failure

Exit code	Meaning
0	Every check passed
1	A check failed, or test stopped before asking
2	A spec has a lint error

Exit 1 has two causes, and the last line of the log tells them apart. Here is the quickstart’s guard tested with --max-cost 0 on a fresh store, before any answers exist:

command_guard: would ask 114 answers in 38 requests (~$0.0010), above --max-cost $0.0; nothing asked

A script can tell them apart without reading the log. A finished test writes its results to .hunch/target/, as JSON with a passed field; a stopped one writes nothing. See the results file.