Test in CI

Set measurable checks in a spec and run them in CI against labelled or reviewed answers.

A unit test checks one expected result. A model judgment needs that kind of check for important examples, plus measurements across rows: how often it disagrees with known answers, and whether its probabilities match the observed outcomes. These measurements need labelled or reviewed rows.

​

Thresholds live in the spec

Add a tests: block, keyed by question:

prototype/examples/swe_agent/patch_eval.yml

tests:
  passes_tests:
    min_auroc: 0.70
    max_calibration_error: 0.10

hunch test checks each configured threshold, prints PASS or FAIL, and exits 1 if a required check fails. A check marked severity: warn prints WARN without failing the build. CI therefore enforces the limits you write; it does not reject every edit that makes any individual answer worse.

Here is a judgment that reads a coding agent’s patch and predicts whether it will pass the project’s tests, from the SWE-agent cookbook, run against the real test results. These are the key lines of the summary:

hunch test prototype/examples/swe_agent/patch_eval.yml --max-cost 0

Test complete · prototype/examples/swe_agent/patch_eval.yml
  Configured check failed
    passes_tests: weighted accuracy 77.8% on 200 gold rows
      AUROC 0.831 · calibration error 0.147
      FAIL max_calibration_error · observed 0.147, limit 0.1
      PASS min_auroc · observed 0.831, limit 0.7

The model ranks patches well: passing patches tend to get higher probabilities than failing ones. Yet the spec fails its calibration limit. On these rows, the model’s probabilities are more confident than the observed outcomes support. A pipeline that skipped tests on confident “yes” answers would risk shipping broken patches. Ranking alone would not reveal that problem.

The other tests are min_accuracy, min_act_accuracy (accuracy among answers above the act bar) and order_stability for choices; the spec reference describes each. A question with no configured threshold still reports measurements, marked measured rather than passing. Its measurements alone cannot fail CI. If a configured check lacks gold, the report marks it not assessed.

Two more kinds of check live in the same spec. Examples check named cases on each run; they do not prove what will happen on unseen rows. Metrics check a rule built from several answers, such as “stop the command if any question says yes”, against gold, with limits like max_missed.

Thresholds compare the point estimate, not the lower end of its interval. With only 30 reviewed rows the uncertainty can be wide; read the interval before choosing a limit.

​

What CI needs

test compares answers with gold, so CI needs both.

Gold is your answer key column and your *.reviews.csv files. Commit them.

Answers come from the store. test asks the engine for any it lacks, which in CI would mean paying on every pull request. Two things keep that small:

• Cache the store between runs. Answers are stored by their exact input, so a restored store is always safe to reuse, and only changed rows or questions are asked again.
• Always pass --max-cost. If the missing answers would cost more, test asks nothing and exits 1.

If anything may be asked, CI also needs an API key: TYPESAFE_API_KEY for Jev, or your engine’s key (Environment).

​

A GitHub Actions workflow

.github/workflows/judgments.yml

name: judgments
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - name: Restore the answer store
        uses: actions/cache@v4
        with:
          path: .hunch/store.sqlite
          key: hunch-store-${{ github.run_id }}
          restore-keys: hunch-store-
      - name: Install hunch
        run: uv tool install hunch-ai
      - name: Test
        env:
          TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
        run: hunch test evals/ --max-cost 0.50

To put the numbers on the pull request itself, add a step that turns the results file into a comment. results_comment.py reads only the published results format, with the standard library; the step fetches it, or copy it into your repository and pin it. The job also needs permissions: pull-requests: write:

      - name: Comment the numbers on the pull request
        if: always()
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          curl -sSfLO https://raw.githubusercontent.com/oneryalcin/hunch/main/prototype/examples/ci/results_comment.py
          python results_comment.py .hunch/target/evals.json > comment.md
          gh pr comment ${{ github.event.pull_request.number }} --body-file comment.md --edit-last --create-if-none

On GitHub Actions, test also adds a table to the run’s summary page: per question, accuracy, its range and any failed check, and a line for each metric and for the examples. The same numbers are in the results file for any other CI.

The cache key changes every run and restore-keys picks the newest store, so answers asked in one run are there for the next. hunch uses the nearest .hunch/store.sqlite at or above the spec’s folder, or creates one at the repository root. Set HUNCH_STORE to choose another path.

To see what a pull request does to the answers, add a step with hunch diff evals/ --against git:origin/main --max-cost 0.50. It reports and does not fail the build.

​

Reading a failure

Exit code	Meaning
0	Every check passed
1	A check failed, or test stopped before asking
2	A spec has a lint error

Exit 1 has two causes, and the last line of the log tells them apart. Here is the command guard tested with --max-cost 0 on a fresh store, before any answers exist:

command_guard: would ask 114 answers in 38 requests (~$0.0012), above --max-cost $0; nothing asked

A script can tell them apart without reading the log. A finished test writes its results to .hunch/target/, as JSON with a passed field; a stopped one writes nothing. See the results file.