CLI

Every hunch command and flag, what each line of output means, and the exit codes.

hunch <command> <path> [flags]
hunch init [--list] RECIPE [DIR]

path is a spec file or a folder of specs. Every command lints the whole project first: warnings go to stderr; an error exits 2.

Commands

Command	What it does	Asks the engine?
lint	Checks the specs and prints lint: ok (N judgments)	No
compile	Shows one real request and what run would ask and cost	No
run	Answers every row, caches the answers, writes one table per judgment	Yes, missing answers only
test	Scores the answers against gold and checks the tests: thresholds	Yes, missing answers only
review	Walks you through the rows that need a human verdict	Yes, missing answers only
diff	Compares this spec with an older or other version on today’s rows	Yes, missing answers only
suggest	Asks an LLM to rewrite one question and keeps a rewrite only if it wins	Yes (writer and new answers)
init	Copies a starter recipe into a folder	No
skill	Installs the skill that teaches a coding agent to use hunch	No
hook	Guards a coding agent’s shell commands: install wires it in, run is what the agent calls	Yes, per command
distill	Trains a small local model per question on the answers already in the store	No
docs	Writes a page people can read and search: each judgment’s status, numbers and lineage	No

Commands ask only for answers the store lacks. --max-cost 0 guarantees no spend: if anything is missing, the command stops before asking. For suggest, the cap also covers each writer call at its worst case.

hunch init

hunch init --list               # the recipes, one line each
hunch init agent-commands       # copies the recipe into ./agent-commands
hunch init agent-eval evals/    # copies into evals/

The folder must not exist. Recipes: agent-commands (a guard for 38 real coding-agent shell commands, the quickstart, about $0.002), agent-eval (check a coding agent’s work from its trace), tickets (40 made-up support tickets: a choice, a yes/no and a score).

hunch skill

hunch skill          # into this folder
hunch skill ~/       # for every project on this machine

Writes the hunch skill to .claude/skills/hunch/SKILL.md (Claude Code) and .agents/skills/hunch/SKILL.md (Codex; Cursor reads both). It tells the agent how to run the loop (lint, compile, run with a cost cap, test, diff) and what not to do: spend without --max-cost, write *.reviews.csv, or edit .hunch/. Each file prints installed, updated or up to date; an existing copy is replaced, so re-run after upgrading hunch and commit the result.

hunch hook

hunch hook install                   # Claude Code and Codex, in this folder
hunch hook install --agent claude    # one agent: claude, codex or both
hunch hook run hunch/command_guard.yml   # what the agent calls, with the event on stdin (--deny for Codex)

install copies the command guard (spec, 38 labelled commands, NOTICE) into hunch/, keeping files already there, adds .hunch/ to .gitignore, and adds a PreToolUse hook with matcher Bash and a 30-second timeout to .claude/settings.json and .codex/hooks.json, keeping the rest of each file. It writes nothing if a settings file can’t be read. The hook command finds the spec through $CLAUDE_PROJECT_DIR (Claude Code) or the git root (Codex), and ends in || echo a warning, so a missing or older hunch lets commands through instead of blocking them.

run reads the agent’s event, judges the command with request (the last message the person typed, from a Claude Code transcript; empty otherwise), cwd, description and command, and logs the row, redacted by the spec’s rules, for --traffic. When any answer is yes or routes to review, it replies ask, or deny with --deny (Codex, which cannot ask yet), with the reasons; otherwise it prints nothing. It never replies allow. On any error it prints a systemMessage warning and lets the command through. HUNCH_MAX_COST defaults to 0.01 per command. See Guard your coding agent.

hunch distill

hunch distill intent.yml                                  # from the answers for the spec's own rows
hunch distill intent.yml --source banking77_unlabeled.csv # or for these rows
hunch distill hunch/command_guard.yml --traffic           # or for rows logged by judge(..., log=True)

Needs the distill extra. Asks nothing: it reads the store’s answers for --node (default: the last judgment) and writes distilled/<judgment>@<hash>/ next to the spec (meta.json and weights.npz). Per question it trains a linear layer on a frozen MiniLM encoder, class-balanced, on the row’s gold where there is some and the engine’s answer otherwise, holding out half the gold rows by a hash of their text. It prints each question’s training size and rarest answer, warning under 30 examples, and the lines to measure and use it. See Distill a decision.

hunch docs

hunch docs support/

Writes two files beside the results file (see below), named after the spec or folder: <name>.html, one self-contained page with no server or network needed, and <name>.manifest.json. It reads the specs, the results of the last test of the same path and the store’s run log, and never asks the engine. See Share what your judgments decide.

The manifest

Field	Meaning
version	Format version (1). Field names are stable within a version.
git_sha, project	The commit, and the spec or folder name. No timestamp, so an unchanged project writes an identical file.
judgments.<name>	The spec as written, every key it sets (a multi question stays one question), plus file (the spec’s path), spec_hash (the hash test records; the page marks a judgment stale when they differ) and upstream (the judgments it reads).

Flags

Flag	Used by	Default	What it does
--version	alone		Prints the installed version and exits.
--node NAME	run, test, diff, review, compile, suggest	all (run, test), required when a project has several judgments (review, suggest)	One judgment in a project. run --node X runs X and the judgments it reads from.
--against REF	diff, review	none	diff: the old version, a spec/folder path or git:REF (for example git:HEAD~1); with --model and no --against, the same specs on their own engine. review: put rows where that version answers differently first
--source CSV	all	the spec’s source	Run the root judgments on this CSV instead (a holdout set, for example)
--traffic	all	off	Run the root judgments on rows logged by judge(..., shadow=...). Not with --source
--model ID	all	the spec’s model	Use this engine for every judgment. Its tables get a suffix (intent__deepseek_deepseek_flash), so the spec’s own tables are untouched
--max-cost USD	all	$HUNCH_MAX_COST, else no cap	The most one set of asks may be charged: a judgment’s missing answers, and separately its escalations and test’s order check. Above the estimate, nothing is asked; while asking, a request is sent only if its worst case still fits, so the charge stays under it
--sample N	compile, run, test, diff	all rows	Judge only N rows of each root source, the same N every time
--allow-change	run	off	Accept a changed spec under on_change: freeze
--list	review	off	Print the queue instead of prompting
--limit N	review	no limit	At most N rows this session
--audit N	review	30	Random rows to spot check per question, counting ones already reviewed
--reviewer NAME	review	your login name	Name saved with each verdict
--question QID	suggest	required if the spec has several questions	The question to rewrite
--n N	suggest	3	Rewrites to try
--writer MODEL	suggest	deepseek:deepseek-flash	The LLM that writes rewrites

--sample picks rows by a hash of the key, so a repeated sample comes from the store and a later full run asks only the rest. run --sample caches its answers but does not replace any table, since downstream readers would take the sample for the whole.

Exit codes

Code	Meaning
0	Done. For test: every check passed
1	test: at least one FAIL line. Any command: stopped with a message (above --max-cost, on_change: freeze refused, unknown judgment)
2	Lint error in a spec, or bad arguments

test exits 1 both when a check fails and when --max-cost stops it from asking (for example, order_stability needs new answers). Read the last line of output to tell them apart.

Reading the output

compile

# intent: request for row 1 (one request per row, all questions read once)
{
  "model": "jev-1.13.0",
  "state": {
    "text": "I still have not received my new card, I ordered over a week ago."
  },
  "questions": { "intent": { "type": "choice", ... } }
}

# intent: 770 rows in; 770 answers planned, 770 cached, 0 to ask, ~0 input tokens, ~$0.00000 (jev-1.13.0)

Line	Meaning
JSON	The exact request for the first row, after redact and clip. For an LLM engine, the prompt.
# <judgment>: …	Rows in, answers planned, cached, to ask, estimated tokens and cost.
where keeps ≤ N, rows kept because the answers their where-clause needs aren't cached yet	Upstream hasn’t run; count and cost are upper bounds.
size warning	A row’s state is near the engine’s limit.
on_change: freeze and the spec changed	run will refuse without --allow-change.
# total	The whole project, with an expected cost when some rows are upper bounds.

run

intent: 385 rows in; answers: 385 cached, 0 asked in 0 requests, 0 input tokens, $0.00000
  intent: 74 rows below act=0.9 → review queue
materialized 1 table(s) in .hunch/store.sqlite (run 2026-09-25T10:45:03-8b2bab)

• First line: answers cached and asked, requests, input tokens, actual cost.
• rows below act: confidence under act; review shows them as “model unsure”.
• The run id is stored on every table row (Store).
• stderr shows progress while asking: asking 529 answers in 529 requests (~$0.0168) and the request rate.

test

Real output, BANKING77 holdout (hunch test intent.yml --source banking77_holdout.csv), trimmed:

intent (choice, 385 rows)
  gold: 385 rows (243 from source, 142 from review, 25 with two acceptable labels)
  PASS estimated accuracy 95.8% (95% CI 88.7%–97.9%) from reviews of 60/340 agreeing rows, 45/45 disagreeing rows (min 85%)
       not the headline: on current gold 97.9% (trusts unreviewed rows), on the raw answer key 88.3%
  PASS calibration error 0.043 (raw source gold: 0.053) (max 0.05)
         stated p          n   avg stated   observed
       0.4–0.5           4        0.450      0.750
       ...
       0.9–1.0         311        0.990      1.000
       dial   automated   wrong among automated
       0.50       99.0%                    1.8%
       ...
       0.90       80.8%                    0.0%  ← act
  PASS accuracy among auto-acted 100.0% on 81% of rows at act=0.9 (min 97%)
       most confident mistakes (8 total; high confidence + wrong = dangerous, or a gold error → hunch review):
         # 560 gold=card_acceptance                  got contactless_not_working 0.89           Are contactless payments enabled on my new card?
       most confused (gold → got):
           1  card_acceptance → contactless_not_working
  PASS order stability: 11/300 answers flip (3.7%, 8 within noise band), mean |Δp| of original answer 0.032 (max flip rate 5%)

answers: 685 cached, 0 asked in 0 requests, 0 input tokens, $0.00000

Line	Meaning
gold:	Rows with a right answer, by origin: source column, reviews, two acceptable labels. Ambiguous and needs-context rows are excluded.
context:	After “needs more context” verdicts: the share of random spot checks that needed more than the state shows.
estimated accuracy	The headline, from reviewed rows, with a 95% interval. With an answer key: all disagreements plus a random sample of agreements, weighted to all rows. Without: random spot checks only. (min …) is the threshold. See How the numbers work.
accuracy	Shown instead when there is no estimate yet: plain accuracy on current gold. upper bound only, no estimate means spot checks are missing.
not the headline	Accuracy on current gold (trusts unreviewed rows) and on the raw answer key.
calibration error	Row-weighted gap between stated p and observed, with the per-band table. Observed above stated: too modest; below: overconfident.
AUROC	Yes/no: how well p(yes) separates yes from no. 0.5 is chance, 1.0 perfect.
dial	Per threshold: share of rows automated and share of those wrong. ← act marks act; yes/no questions get one column pair per side. Computed on rows with gold.
accuracy among auto-acted	Accuracy on rows at or above act.
most confident mistakes	Wrong answers, most confident first: a model error or a gold error.
most confused	Most common (gold → got) pairs.
order stability	With the order_stability test: flips when options are shuffled; noise-band flips (margin under 0.10) are expected.
declined (none_of_these), escalated to …	Questions with none or escalate.
weighted to the population	The source has weights; every number is weighted.
last line	Answers cached and asked, and cost.

The results file

When test finishes, it also writes what it printed as JSON, for CI, dashboards and agents: .hunch/target/<tested path>.json, beside the store, with the path of the tested spec or folder relative to the store’s folder (hunch test examples/tickets/triage.yml writes .hunch/target/examples/tickets/triage.json). With --model, the engine’s suffix is added, as for its tables (triage__deepseek_deepseek_flash.json), so another engine’s results never replace the spec’s own. test removes the file when it starts, so a run stopped by --max-cost leaves none rather than an old one.

Field	Meaning
version	Format version (1). Field names are stable within a version.
command, at, git_sha	test, when it ran (UTC), the commit it ran on.
passed	Every check passed, or failed only with severity: warn (the exit code was 0).
sample	N if test ran with --sample N, else null.
cost	Spent by this run, in USD.
judgments.<name>	spec_hash, model, rows, and questions (and multi, for type: multi).
questions.<q>.gold	Rows with gold: rows, and by origin source, review, excluded.
questions.<q>.accuracy	value, ci ([low, high], or null when basis is gold: plain accuracy against gold, no estimate), basis (estimate or gold), reviewed (per group: reviewed of of rows).
questions.<q>.calibration_error, auroc	As printed; auroc for yes/no questions with both answers in gold.
questions.<q>.act	With act in the spec: threshold, automated (share of all rows acted on), judged (acted-on rows with gold), wrong (share of those that are wrong; null if none has gold).
questions.<q>.dial	Per threshold: automated and wrong (shares); yes/no questions have them under yes and no.
questions.<q>.mistakes	total, and most_confident: id, got, p, gold.
questions.<q>.checks	Each check: check (the test key, like min_accuracy), passed, value, limit, severity.
questions.<q>.spot_checks, needs_context	Random spot checks, and how many of them needed more context.
judgments.<name>.metrics.<m>	rule, rows, fired, rate; with gold, gold (basis, rows), and gold_rate, missed, false_alarms (each count, of, rate, ci); checks. See metrics.
judgments.<name>.examples	Per example: name, passed, and answers (question, expected, got, p, passed). See examples.

diff

Real output, BANKING77 (hunch diff intent.yml --against intent_bare.yml), trimmed:

answers: 1540 cached, 0 asked in 0 requests, 0 input tokens, $0.00000

══ intent  vs  intent_bare

intent: 77/770 rows flip (15 within noise band)
  probabilities moved: mean |Δ| 0.102, mean Δ p(old answer) -0.021
  gold accuracy on the 770 shared rows with gold 79.0% → 85.5%  (✓ 58 fixed, ✗ 8 broken, 11 wrong both times, 0 without gold)
  paired sign test p=0.000 → significant
  ✓ #  76                activate_my_card 0.92 → card_linking 0.84                           Could you help me reactivate my card? I…
  ✗ # 255           fiat_currency_support 0.66 → exchange_via_app 0.91                       How do I change currencies to euros?
    # 268                   exchange_rate 0.68 → exchange_via_app 0.94                       I want to make a currency exchange to E…
  … 65 more

Line	Meaning
rows flip	Rows whose answer changed; within noise band counts flips where either side’s margin is under 0.10.
N rows newly reach it, M no longer do	An upstream where changed which rows this judgment sees.
probabilities moved	mean |Δ|: size of the move. mean Δ p(old answer): direction (for yes/no, the change in p(yes)).
gold accuracy … →	Old and new accuracy on shared rows with gold: fixed, broken, wrong both times, without gold.
paired sign test	Whether fixed vs broken beats chance.
flip rows	✓ fixed, ✗ broken, blank: no gold or wrong both times. ~noise flips are listed last.
unchanged (same keys, 0 calls), new question, renamed from 'x', moved by upstream changes	Other outcomes; renames are matched by answer keys.
affects …: N rows	Per exposure, the rows whose change it can see: an answer changed, appeared or disappeared on a question it reads, and, if its uses names the answers it relies on, one of those was the old or the new answer.
not affected: …	Exposures that read a question in the diff but see no change.

Both sides use today’s reviews.

review

See Review for the screen and keys. --list prints one row per item:

review queue: 112 disputed, 30 audit
  disputed  #  281 intent     card_arrival 1.00                    gold=card_delivery_estimate   my card was not in the mail again can you advise?

Columns: kind, row id, question, model answer and confidence, answer key (- if none), start of the row.

Verdicts saved to <judgment>.reviews.csv (columns qid,row_id,state_hash,verdict,label,reviewer,at,kind):

Verdict	Saved when
model_right / key_right	On a disputed row, you picked the model’s answer / the answer key
against_right / spec_right	On a shadow row, you picked the --against spec’s answer / this spec’s
confirmed	On a spot check, you picked the marked answer
labeled	You picked any other option
both_ok	b: two answers are acceptable; both count as right
ambiguous	a: no single right answer; left out of scoring
needs_context	c: the text shown is not enough to judge; left out of scoring and counted in test’s context line

kind says why the row was reviewed: disputed, audit (a random spot check), uncertain, shadow, or any other name you write when importing verdicts (for example flagged). Only audit rows feed the accuracy estimate; every verdict counts as gold. test also marks a row same_text when it takes its gold from another row with identical text: gold, but not a spot check.