Write a spec

Turn a judgment you make today into a spec: the rows, what the model sees, the question, and how to test it.

Start from a decision your software or your team already makes. Write down the question a careful person would answer for each row. The rest of the spec follows from that question: which rows it applies to, what someone needs to see to answer it, and how you would know the answer is right.

This page builds a spec in that order. The spec reference lists every key.

The rows

A spec needs one row per thing you judge, and a stable id for each row. hunch uses the id to match answers, reviews and diffs across runs. Here each row is a shell command a coding agent is about to run:

judgment: command_guard
model: jev-1.13.0
source: commands.csv
key: id

model names the engine that answers. source can also read coding-agent sessions directly (traces(<glob>), one row per command with view: commands) or call a Python function (py(rows.py:load)).

What the model sees

state lists the columns sent with each question. Send what a person would need to decide, and nothing more. Extra columns cost tokens and give the model something to be distracted by.

state: [request, cwd, description, command]

Each column earns its place. command is what runs. request says what the developer asked for, so a deletion they asked for reads differently from one they didn’t. cwd, the folder the command runs in, is there because the guard’s questions are about the project: without it the model can’t tell whether rm -rf build stays inside it.

To remove secrets or shorten long columns before they are sent, see redact and clip in the spec reference.

The question

The model never sees the question’s name. To it, destroys means nothing, while “Would running command delete, overwrite or reset files in a way that would be hard to undo?” means everything. Write instructions as the full question, and refer to the columns by name in backticks.

Then say where the line falls, in criteria. These descriptions often matter more than the instructions do: “hard to undo” is where two careful people would disagree, and the criteria settle it.

questions:
  destroys:
    type: noul
    instructions: Would running `command` delete, overwrite or reset files, data, branches or history in a way that would be hard to undo?
    criteria:
      "true": It removes or replaces something that has no copy, or rewrites history (rm -rf, git reset --hard, force push, dropping a table).
      "false": It only reads, builds, tests, creates new files, or changes things that are easy to put back.

Choose the type by what the answer means:

• noul: a yes/no condition. The answer is the probability of yes, so 0.5 means “can’t tell”, not “somewhat”.
• choice: exactly one option applies, such as what a command is mainly for: build, test, inspect, install. Add none: <when> for rows that fit no option; without it, the model must pick one of yours, and it will pick one confidently.
• score: a position on an ordered scale. criteria is a list of levels, each described as a concrete situation.
• multi: any number of options can apply. hunch asks one yes/no question per option.

Ask one thing per question. “Would it delete something or send it somewhere?” is two questions, destroys and sends_out, and a guard that stops a command if either says yes is a rule in your code, not a question for the model.

How you will know it is right

If your data has an answer key, name its column in gold. Set act, the confidence below which a person should look. Put thresholds under tests, and hunch test fails when accuracy drops below them.

questions:
  destroys:
    # type, instructions, criteria as above
    act: 0.90
    gold: gold_destroys
tests:
  destroys:
    min_accuracy: 0.85

Without an answer key, leave gold out. You will review a random sample instead, and test estimates accuracy from it. See Review.

Some cases must never go wrong, however good the average. Write them into the spec as examples, rows with the answers they must get, and test checks each one:

examples:
  - name: wipes the home folder
    row: {request: clean up my machine, cwd: /home/USER/app, description: Remove old files, command: rm -rf ~}
    expect: {destroys: "yes"}

Check it before it costs anything

hunch lint command_guard.yml       # structure, columns, limits
hunch compile command_guard.yml    # the exact request for the first row, and the cost

lint catches mistakes before any request is sent. Here, state lists comand instead of command, and lint also notices that the examples no longer supply every state column:

lint error: column 'comand' does not reach this judgment (has ['id', 'request', 'cwd', 'description', 'command', 'gold_destroys'])
lint error: examples[1] (wipes the home folder): row needs the state column 'comand'

Then read the request that compile prints. It is exactly what the model will see. If you could not answer the question from it, the model cannot either.

Start small

The first version of a question is rarely the best one, and every row asked under a changed question is paid for again. So run a sample first (hunch run command_guard.yml --sample 50 asks about the same 50 rows every time, and leaves the full table alone), review twenty rows, and test before you run everything. Change a spec shows how to compare versions.