Concepts

The ideas hunch is built from, each resting on the one before: rows, questions, specs, the store, gold, act and review.

A command, a support message and a model answer look different, but hunch handles them in the same order: choose a row, ask a question, compare answers with evidence, then decide which answers need a person.

​

A row is one thing you judge

A row is one shell command, one agent turn, or one product review, with an id named by the spec’s key. Rows come from a CSV file, coding-agent session logs (traces(...)), or a Python function (py(file.py:fn)).

​

A question asks one thing of every row

A question has instructions, which say what is being asked, and usually criteria, which describe the possible answers. The model sees these together with the row’s state, the columns you chose to show it. It never sees the question’s name, so the instructions must carry the whole meaning.

The answer is always typed, never free text. The type decides its shape:

Type	The answer	For example
choice	one option, with a probability for each	how an agent’s turn went: worked, failed, redirected, unclear
noul (yes/no)	the probability that the answer is yes	would this command destroy something hard to undo
score	a level on a described scale	how risky a change is, from harmless to irreversible
multi	yes or no for each option, asked as one noul per option	which kinds of side effect a command has

​

A spec writes the questions down

A spec is a YAML file for one judgment. It says where the rows come from, what the model sees, which questions to ask, and which tests the answers must pass. It lives in git beside your code.

A folder of specs is a project, where one judgment can read another’s results.

​

Every answer is stored under its exact input

When hunch asks a question, it stores the answer in .hunch/store.sqlite. Its cache key covers the exact request: the model, the state, and the question as sent. A later run can reuse an answer while that request and store remain the same.

Here is what goes into that key, and what stays out:

model: jev-1.13.0             # in the key
state: [request, command]     # in the key: these columns' values, after redact and clip
questions:
  destroys:                   # the name: not sent, not in the key
    type: noul                # in the key
    instructions: Would running `command` delete something in a way that is hard to undo?   # in the key
    criteria: {...}           # in the key
    act: 0.90                 # not sent, not in the key
    gold: gold_destroys       # not sent, not in the key
tests: {...}                  # not sent, not in the key

This one rule explains much of how hunch behaves:

• A re-run with nothing changed uses the stored answers and asks nothing.
• Changing one question asks only that question again.
• Old answers stay in the store, so diff can compare two versions of a spec without asking anything.

​

Gold is the answer you trust

To measure how often the model is right, you need rows with an answer to compare against. hunch calls these answers gold. They come from an answer-key column in your data (gold:), or from your verdicts in review. The measured rate describes those rows; how well it applies to future rows depends on how representative they are.

An answer key is someone else’s judgment, and it can be wrong (one public key was wrong four times as often as the model). So hunch treats a key as a starting point: the rows where the model and the key disagree are the first ones it asks you about.

​

act sets a bar for review

An answer with probability 0.99 clears a higher bar than one with 0.55, but the number alone does not prove that either answer is right. act is the confidence below which an answer should go to a person instead of being acted on automatically. Your application or hook uses that bar to choose what happens next.

Choosing act is a trade: raising it sends more rows to review. On labelled or reviewed rows, hunch test shows the share above each threshold and the mistakes in that share. Neither act nor gold is sent to the model, so changing them asks nothing.

Here is that trade on 770 real answers from the banking cookbook. Each dot is one message, the most confident at the top. Move act and watch mistakes leave the automated block and land with a person:

At 0.90, 82.6% of these answers clear the bar; 8.2% of that group disagree with the answer key. Raising the bar moves rows from the automated group to review. Some red dots near the top mark errors in the answer key rather than the model, which is why disagreements deserve review.

​

Review turns your verdicts into gold

hunch review shows you rows one at a time and records your verdict. It orders them by how much each verdict tells hunch:

1. rows where the model and the answer key disagree
2. a random sample of the rest
3. rows below act

The random sample matters. If you only reviewed disagreements, every verdict could only move the estimate up. Random rows let test estimate accuracy honestly, with a range around it.

A verdict belongs to the exact text you judged. If the row’s text changes, the old verdict no longer counts, and the row comes back to the queue.

​

An engine answers the questions

The examples use TypeSafe’s Jev, a model built to return typed answers with probabilities. Any LLM that exposes token probabilities also works. See Engines.

​

How the pieces meet

The spec and row go to an engine, which returns an answer and probability. When a question sets act, its bar routes the answer toward application code or review. Without act, the route is empty and your code decides what to do. Reviewed answers become gold for hunch test.

In practice, run a spec, review a sample, test, then change the spec and diff to see what flipped. Your application can call the same spec with hunch.judge(...).