Grade Claude Code sessions

Find out whether a coding agent’s turn worked by reading what the developer wrote next.

A developer asks a coding agent to make the background a darker purple blue in dark mode. The agent works for a while and ends its turn:

Light mode keeps the existing blue gradient, dark mode switches to a deep purple-to-dark-navy (#1a0533 → #0d1b4b).

It sounds done. Then the developer writes: “background did not change”.

Nobody labelled that turn, and nobody had to. The next message said how it went. How would your code read it, across hundreds of turns a week?

Anyone running coding agents wants to know how often a turn works, and nobody has time to label turns. The label is already in the session, written in plain words by the developer. Reading those words is a judgment, and that’s the part hunch is for.

​

Try it first

These are real turns from public Claude Code sessions. Read what the developer wrote next and say how the turn went. Then see what hunch said, how sure it was, and what three reviewers decided.

Some turns read themselves. Others take judgment, and the reviewers and hunch can disagree. What matters is that hunch says how sure it is, so the hard turns can go to a person.

​

1. Read sessions as rows

A spec can read agent session files directly. Claude Code, Codex, Cursor, OpenCode and OpenTelemetry GenAI files are recognised by their contents:

# a glob, relative to the spec; ~ works
source: traces(sessions/*/*.jsonl)
view: turns   # a row per message typed

A turn is one message a person typed and everything the agent did until the next one. Each row has request (what they asked), final_reply (the agent’s last message for that turn) and next_message (what they wrote next), plus counts of tool calls and edits. Text the harness injects, such as IDE context and system reminders, is removed. The other views and columns are in the spec reference.

Look at the rows before writing anything:

uvx --from hunch-ai python -m hunch.traces \
  'sessions/*/*.jsonl' turns

​

2. Keep private things private

Sessions hold paths, emails, keys and whatever else was pasted into them. redact removes these from every state column before a row is hashed, sent or logged. clip keeps long fields to a size:

redact: [secrets, emails, home,
         "[0-9]{1,3}([.][0-9]{1,3}){3}"]
clip: {request: 3000, final_reply: -3000,
       next_message: 1500}

secrets, emails and home are built in, and the quoted pattern catches IPv4 addresses. A negative number keeps the end of a field: the last 3,000 characters of the reply, where an agent says what it did. hunch compile prints the exact request for the first row, which is what will leave your machine.

​

3. Ask how the turn went

The state list is what the model may read. The question is a choice: one of four answers, each with its meaning written out, and how sure it is:

outcome.yml

# the example's file, without its exposures
judgment: outcome
model: jev-1.13.0
source: traces(.cache/trace-commons/sessions/claude_code/*.jsonl)
redact: [secrets, emails]
clip: {request: 3000, final_reply: -3000, next_message: 1500}
key: id
state: [request, final_reply, next_message]
questions:
  outcome:
    type: choice
    instructions: >-
      A developer asked a coding agent for
      something (`request`). `final_reply` is the
      agent's last message for that turn and
      `next_message` is what the developer wrote
      next. Judging mainly from `next_message`,
      how did the agent's turn go?
    criteria:
      worked: The developer accepts the result or
        moves on (thanks, approval, a next step
        that builds on it, a new unrelated task)
      failed: The developer reports it did not
        work (the same or a new error, a bug,
        wrong output, "still broken", or pastes
        an error log)
      redirected: The developer says the agent
        misunderstood, took the wrong approach,
        or should stop or undo; an interruption
        counts
      unclear: The next message does not show
        whether the turn worked
    act: 0.80   # below this, a person decides

act is the bar. An answer at least 0.80 sure is used as it is; below that, the turn waits in a queue for a person.

​

4. Ask whether the agent said it was done

A second judgment reads only the agent’s final reply, so it can run the moment a turn ends. noul is a yes or no, with how sure:

claims.yml

judgment: claims
model: jev-1.13.0
source: traces(.cache/trace-commons/sessions/claude_code/*.jsonl)
redact: [secrets, emails]
clip: {request: 3000, final_reply: -3000}
key: id
state: [request, final_reply]
questions:
  claims_done:
    type: noul
    instructions: >-
      In `final_reply`, does the agent state or
      clearly imply that what the developer asked
      for in `request` is done and working? A
      plan, a question back, a partial result, or
      a report that it could not do it is "no".
    act: 0.80

Together the two answer a sharper question: when the agent says it is done, is it?

​

5. Run both and join them

Each run writes a table to the store, .hunch/store.sqlite, and every answer is kept, so running again is free:

cd prototype/examples/claude_code
hunch run outcome.yml --max-cost 1
hunch run claims.yml --max-cost 1
sqlite3 -header -column \
  ../../.hunch/store.sqlite \
  "select c.claims_done, o.outcome, count(*) n
   from claims c join outcome o using(id)
   group by 1, 2"

claims_done  outcome     n
-----------  ----------  ---
no           failed       30
no           redirected   19
no           unclear      10
no           worked       50
yes          failed       45
yes          redirected   13
yes          unclear       3
yes          worked      139

Of the 200 turns where the agent said it was done, the developer’s next message reported a failure or a redirect in 58.

​

The result

A panel of three AI reviewers graded 60 random turns without seeing hunch’s answers. hunch agreed with them on 54:

Of the 46 turns it was sure enough to label on its own, one was wrong: the taskbar icon in the wrong place, from the puzzle above.

​

A clearer wording that did worse

The maintainer ran the same question on their own sessions, and its remaining mistakes had a pattern. The developer’s next message was a follow-up question, and the model read it as failed or worked when it should have said unclear.

The fix looked obvious. failed gained “A follow-up question (asking to explain, compare or recommend) is not a failure”, and unclear gained “such as a follow-up question about the answer”. It reads better. It is in the example as candidates/outcome.yml.

Before shipping it, hunch diff asked the new wording about every turn and compared. On the maintainer’s sessions it changed the answer on more than a fifth of them, fixed none of the graded turns and broke five. On the public turns you can run the same check for free, from the same folder, since every answer is cached:

hunch diff candidates/outcome.yml \
  --against outcome.yml

outcome: 19/309 rows flip (9 within noise band)
  gold accuracy on the 60 shared rows
    with gold 90.0% → 86.7%
    (✓ 0 fixed, ✗ 2 broken,
    1 wrong both times, 16 without gold)
  ✗ #d4c4d47d#15
      redirected 0.69 → unclear 0.60
      2nd component that looks bad on mobile …

One of the two it broke is the mobile layout turn from the puzzle. “Wouldn’t it just make sense to make it vertical” is a change of approach, and the new wording had taught the model that a question means unclear.

The old wording stayed. A sentence meant for a handful of rows moved many more, and only asking it about every row showed that.

​

Use it on your data

Point source at your own session files, run outcome.yml, review a few dozen turns, and hunch test tells you how often it reads your developers right. When the sessions are yours, grade from the text on screen, not from what you remember happened next: press c (needs more context) when only your memory decides a turn.

​

How it was measured

The data

28 public Claude Code sessions from Trace Commons agent traces (CC BY 4.0): real developers working with coding agents on public repositories, anonymized and reviewed by each contributor. They split into 309 turns. prototype/examples/claude_code/NOTICE.md has the download command; nothing from the sessions is committed except the excerpts of 60 turns the reviewers saw, in prototype/review_panel/claude_code/packets/. Both judgments together cost $0.02. outcome answered 189 worked, 75 failed, 32 redirected and 13 unclear. The results table in .hunch/store.sqlite keeps rows as they were read, unredacted, so keep .hunch/ out of git.

The review panel

Three AI reviewers (one Claude Opus, two Claude Sonnet, one of them reading in reverse order) graded 60 turns chosen by hash, with the same definitions and the same redacted, clipped text as the model, and none of the model’s answers. Every turn had a majority. They agreed with each other at Fleiss kappa 0.88 on outcome (53 of 60 unanimous) and 0.86 on claims_done (54 of 60). The majority labels are outcome.reviews.csv and claims.reviews.csv; prototype/review_panel/claude_code/ has the packets, the answers and the scripts.

Accuracy, with its interval

From hunch test, against the panel:

outcome (choice, 309 rows)
  gold: 60 rows
    (0 from source, 60 from review)
  PASS estimated accuracy 90.0%
    (95% CI 79.9%–95.3%)
    from reviews of 60/309 random rows

claims_done scored 88.3% (95% CI 77.8–94.2%); of its seven mistakes, five said no where the panel said yes. At act: 0.80, outcome labels 76.7% of the graded turns on its own with 2.2% of those wrong, and sends 75 of the 309 turns to the review queue (claims_done: 81). Its mistakes lean one way: 5 of its 6 said “worked”, and in 4 of those the panel saw a failure or a redirect. So failure rates read from these answers are, if anything, low.

Does running something after the last edit matter?

Each turn also records whether the agent ran a command after its last edit (ran_after_edit). An earlier reading of these sessions found that, among turns where the agent edited files and said it was done, the developer reported a problem in 33% of the 142 turns where it ran nothing after its last edit, against 16% of the 31 where it ran a command; resampling by session put the gap between −1 and +30 points. That reader did not count PowerShell as running a command. The current one does, and the gap is gone: 30% of 115 turns against 28% of 61, and resampling the 23 sessions puts the gap between −14 and +16 points. With PowerShell left out again, it comes back (32% of 145 against 16% of 31). On these sessions, the finding does not hold.

The maintainer's own sessions

The next test was the maintainer’s own Claude Code sessions: personal and scratch projects, with work repositories left out. Only aggregate numbers are recorded; no text from those sessions appears here. The first run read 533 turns for $0.04. A later fix to the trace reader stopped it from taking another agent session’s message for the developer’s, which left 475 turns. The spec was the one above with act lowered to 0.60. The developer who wrote the sessions graded random turns from the text on screen. outcome was right on an estimated 91.7% of turns (95% CI 74.2–97.7%), from 24 random turns. At act: 0.60, 58.3% of the graded turns would be labelled automatically, and none of those were wrong. 1 of 30 random turns needed more of the session than the request, reply and next message. When the agent said it was done, the next message reported a failure or a redirect in 3% of turns (7 of 227); when it did not, 8% (26 of 306). These rates rest on claims answers nobody has reviewed yet.

The wording change, in full

On the maintainer’s sessions, the remaining mistakes were follow-up questions read as failed or worked rather than unclear, with confidence between 0.43 and 0.57. The candidate wording changed the answer on 108 of 475 turns, most from worked to unclear. On the 21 graded turns both versions shared, accuracy fell from 95.2% to 71.4%: 0 fixed, 5 broken. On the public turns it flipped 19 of 309 (9 within the noise band) and moved accuracy on the 60 graded turns from 90.0% to 86.7%. With 0 fixed and 2 broken, the paired sign test gives p = 0.500: on the public turns alone, the drop could be noise. The other broken public turn is a pasted build error, which the old wording read as failed (0.55) and the new one as worked (0.50).

What is still open

claims has not been reviewed on the maintainer’s sessions. The own-session grades came from the person who wrote the sessions, and the review screen shows the model’s answer, so they are not blind. 24 random grades give a wide interval; about 30 more are needed before judging the next wording change. The review panel is not people: it is three AI reviewers reading the same excerpts as the model.