Grade Claude Code sessions

Read coding-agent sessions as rows, judge how each turn went, and check the judge against reviews.

You ask a coding agent for something, it works for a while, and it says it is done. Was it? Across hundreds of turns a week, nobody labels them. Yet every session already holds a clue: the next thing the developer wrote. “Thanks, now do X” means the turn worked. “Still broken” means it did not.

That next message is the closest thing to free gold a conversation has. This cookbook turns sessions into rows, asks a model to read that message, and then checks how often the model reads it right.

Sessions become rows

A spec can read agent sessions directly:

source: traces(sessions/*/*.jsonl)    # a glob, relative to the spec; ~ works
view: turns                           # or runs: one row per session

Claude Code, Cursor, OpenCode and OpenTelemetry GenAI files are recognised by their contents. The turns view gives one row per message a person typed, with request (what they asked), final_reply (the agent’s last message for that turn) and next_message (what they wrote next), plus counts of tool calls and edits. Text the harness injects, such as IDE context and system reminders, is removed. The full list of views and columns is in the spec reference.

Look at the rows before writing a spec:

uvx --from hunch-ai python -m hunch.traces 'sessions/*/*.jsonl' turns

Keep private things private

Sessions contain paths, emails, keys and whatever else was pasted into them. redact removes these from every state column before a row is hashed, sent or logged. clip keeps long fields to a size.

redact: [secrets, emails, home, "[0-9]{1,3}([.][0-9]{1,3}){3}"]   # built-ins, plus IPv4 addresses
clip: {request: 3000, final_reply: -3000, next_message: 1500}   # -N keeps the last N characters

hunch compile prints the exact request for the first row: that is what will leave your machine. The judgment’s results table in .hunch/store.sqlite keeps rows as they were read, unredacted, so keep .hunch/ out of git.

Two questions about each turn

outcome reads the next message. claims reads only the agent’s final reply, so it can run the moment a turn ends. Together they answer a sharper question: when the agent says it is done, is it?

outcome.yml

judgment: outcome
model: jev-1.13.0
source: traces(sessions/*/*.jsonl)
redact: [secrets, emails, home]
clip: {request: 3000, final_reply: -3000, next_message: 1500}
key: id
state: [request, final_reply, next_message]
questions:
  outcome:
    type: choice
    instructions: >-
      A developer asked a coding agent for something (`request`). `final_reply` is the agent's last message
      for that turn and `next_message` is what the developer wrote next. Judging mainly from `next_message`,
      how did the agent's turn go?
    criteria:
      worked: The developer accepts the result or moves on (thanks, approval, a next step that builds on it, a new unrelated task)
      failed: The developer reports it did not work (the same or a new error, a bug, wrong output, "still broken", or pastes an error log)
      redirected: The developer says the agent misunderstood, took the wrong approach, or should stop or undo; an interruption counts
      unclear: The next message does not show whether the turn worked
    act: 0.80

claims.yml

judgment: claims
model: jev-1.13.0
source: traces(sessions/*/*.jsonl)
redact: [secrets, emails, home]
clip: {request: 3000, final_reply: -3000}
key: id
state: [request, final_reply]
questions:
  claims_done:
    type: noul
    instructions: >-
      In `final_reply`, does the agent state or clearly imply that what the developer asked for in `request`
      is done and working? A plan, a question back, a partial result, or a report that it could not do it is "no".

Run both, then join their tables in the store:

hunch run . --max-cost 1
sqlite3 -header -column .hunch/store.sqlite \
  "select c.claims_done, o.outcome, count(*) n from claims c join outcome o using(id) group by 1, 2"

Public sessions: about 90% right

The first test used 28 public Claude Code sessions from Trace Commons (CC BY 4.0), which split into 309 turns. The specs are in prototype/examples/claude_code/, as shown above with act: 0.80. Both judgments cost $0.02.

A panel of three AI reviewers graded 60 random turns without seeing the model’s answers. Against their verdicts:

outcome (choice, 309 rows)
  gold: 60 rows (0 from source, 60 from review)
  PASS estimated accuracy 90.0% (95% CI 79.9%–95.3%) from reviews of 60/309 random rows

claims_done scored 88% the same way. The model’s outcome mistakes leaned one way: 5 of 6 said “worked” where the panel saw a failure or a redirect.

The join raised a question of its own. Among turns where the agent edited files and said it was done, the developer reported a problem in 33% of the 142 turns where it ran nothing after its last edit, against 16% of the 31 where it ran a command. Turns cluster within sessions, though. Resampling by session puts the gap between −1 and +30 points, so it is suggestive, not proven.

The maintainer’s own sessions

The next test was the maintainer’s own Claude Code sessions: personal and scratch projects, with work repositories left out. Only aggregate numbers are recorded; no text from those sessions appears here.

The first run read 533 turns for $0.04. A later fix to the trace reader stopped it from taking another agent session’s message for the developer’s, which left 475 turns. The spec was the one above with act lowered to 0.60. The developer who wrote the sessions graded random turns from the text on screen:

• outcome was right on an estimated 91.7% of turns (95% CI 74.2–97.7%), from 24 random turns.
• At act: 0.60 (not the example’s 0.80), 58.3% of the graded turns would be labelled automatically, and none of those were wrong.
• 1 of 30 random turns needed more of the session than the request, reply and next message.
• When the agent said it was done, the next message reported a failure or a redirect in 3% of turns (7 of 227). When it did not, 8% (26 of 306). These rates rest on claims answers nobody has reviewed yet.

A better wording that was worse

The remaining mistakes had a pattern: the developer’s next message was a follow-up question, and the model read it as failed or worked rather than unclear, with confidence between 0.43 and 0.57.

The obvious fix was to say so in the spec. failed gained “A follow-up question (asking to explain, compare or recommend) is not a failure”, and unclear gained “such as a follow-up question about the answer”. Before shipping it, hunch diff compared the new wording with the old on the same 475 turns:

• 108 of 475 turns changed answer. Most moved from worked to unclear.
• On the 21 graded turns both versions shared, accuracy fell from 95.2% to 71.4%: 0 fixed, 5 broken.

A line meant to rescue a handful of rows had moved more than a fifth of them. The old wording stayed. This is what diff is for: the change read well, and only the rows showed it was wrong.

Grade from the text, not from memory

When you review your own sessions, you remember what happened next and the model does not. Press c (needs more context) when only your memory decides a row; see Review.

What is still open

• claims has not been reviewed on the maintainer’s sessions.
• The own-session grades came from the person who wrote the sessions. The review screen shows the model’s answer, so the grades are not blind.
• 24 random grades give a wide interval. About 30 more are needed before judging the next wording change.