> ## Documentation Index > Fetch the complete documentation index at: https://fuguai.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Grade Claude Code sessions > Read coding-agent sessions as rows, judge how each turn went, and check the judge against reviews. You ask a coding agent for something, it works for a while, and it says it is done. Was it? Across hundreds of turns a week, nobody labels them. Yet every session already holds a clue: the next thing the developer wrote. "Thanks, now do X" means the turn worked. "Still broken" means it did not. That next message is the closest thing to free gold a conversation has. This cookbook turns sessions into rows, asks a model to read that message, and then checks how often the model reads it right. ## Sessions become rows A spec can read agent sessions directly: ```yaml theme={null} source: traces(sessions/*/*.jsonl) # a glob, relative to the spec; ~ works view: turns # or runs: one row per session ``` Claude Code, Cursor, OpenCode and OpenTelemetry GenAI files are recognised by their contents. The `turns` view gives one row per message a person typed, with `request` (what they asked), `final_reply` (the agent's last message for that turn) and `next_message` (what they wrote next), plus counts of tool calls and edits. Text the harness injects, such as IDE context and system reminders, is removed. The full list of views and columns is in the [spec reference](/reference/spec#view). Look at the rows before writing a spec: ```sh theme={null} uvx --from hunch-ai python -m hunch.traces 'sessions/*/*.jsonl' turns ``` ## Keep private things private Sessions contain paths, emails, keys and whatever else was pasted into them. `redact` removes these from every `state` column before a row is hashed, sent or logged. `clip` keeps long fields to a size. ```yaml theme={null} redact: [secrets, emails, home, "[0-9]{1,3}([.][0-9]{1,3}){3}"] # built-ins, plus IPv4 addresses clip: {request: 3000, final_reply: -3000, next_message: 1500} # -N keeps the last N characters ``` `hunch compile` prints the exact request for the first row: that is what will leave your machine. The judgment's results table in `.hunch/store.sqlite` keeps rows as they were read, unredacted, so keep `.hunch/` out of git. ## Two questions about each turn `outcome` reads the next message. `claims` reads only the agent's final reply, so it can run the moment a turn ends. Together they answer a sharper question: when the agent says it is done, is it? ```yaml outcome.yml theme={null} judgment: outcome model: jev-1.13.0 source: traces(sessions/*/*.jsonl) redact: [secrets, emails, home] clip: {request: 3000, final_reply: -3000, next_message: 1500} key: id state: [request, final_reply, next_message] questions: outcome: type: choice instructions: >- A developer asked a coding agent for something (`request`). `final_reply` is the agent's last message for that turn and `next_message` is what the developer wrote next. Judging mainly from `next_message`, how did the agent's turn go? criteria: worked: The developer accepts the result or moves on (thanks, approval, a next step that builds on it, a new unrelated task) failed: The developer reports it did not work (the same or a new error, a bug, wrong output, "still broken", or pastes an error log) redirected: The developer says the agent misunderstood, took the wrong approach, or should stop or undo; an interruption counts unclear: The next message does not show whether the turn worked act: 0.80 ``` ```yaml claims.yml theme={null} judgment: claims model: jev-1.13.0 source: traces(sessions/*/*.jsonl) redact: [secrets, emails, home] clip: {request: 3000, final_reply: -3000} key: id state: [request, final_reply] questions: claims_done: type: noul instructions: >- In `final_reply`, does the agent state or clearly imply that what the developer asked for in `request` is done and working? A plan, a question back, a partial result, or a report that it could not do it is "no". ``` Run both, then join their tables in the store: ```sh theme={null} hunch run . --max-cost 1 sqlite3 -header -column .hunch/store.sqlite \ "select c.claims_done, o.outcome, count(*) n from claims c join outcome o using(id) group by 1, 2" ``` ## Public sessions: about 90% right The first test used 28 public Claude Code sessions from [Trace Commons](https://huggingface.co/datasets/trace-commons/agent-traces) (CC BY 4.0), which split into 309 turns. The specs are in `prototype/examples/claude_code/`, as shown above with `act: 0.80`. Both judgments cost \$0.02. A panel of three AI reviewers graded 60 random turns without seeing the model's answers. Against their verdicts: ``` outcome (choice, 309 rows) gold: 60 rows (0 from source, 60 from review) PASS estimated accuracy 90.0% (95% CI 79.9%–95.3%) from reviews of 60/309 random rows ``` `claims_done` scored 88% the same way. The model's `outcome` mistakes leaned one way: 5 of 6 said "worked" where the panel saw a failure or a redirect. The join raised a question of its own. Among turns where the agent edited files and said it was done, the developer reported a problem in 33% of the 142 turns where it ran nothing after its last edit, against 16% of the 31 where it ran a command. Turns cluster within sessions, though. Resampling by session puts the gap between −1 and +30 points, so it is suggestive, not proven. ## The maintainer's own sessions The next test was the maintainer's own Claude Code sessions: personal and scratch projects, with work repositories left out. Only aggregate numbers are recorded; no text from those sessions appears here. The first run read 533 turns for \$0.04. A later fix to the trace reader stopped it from taking another agent session's message for the developer's, which left 475 turns. The spec was the one above with `act` lowered to 0.60. The developer who wrote the sessions graded random turns from the text on screen: * `outcome` was right on an estimated **91.7%** of turns (95% CI 74.2–97.7%), from 24 random turns. * At `act: 0.60` (not the example's 0.80), 58.3% of the graded turns would be labelled automatically, and none of those were wrong. * 1 of 30 random turns needed more of the session than the request, reply and next message. * When the agent said it was done, the next message reported a failure or a redirect in 3% of turns (7 of 227). When it did not, 8% (26 of 306). These rates rest on `claims` answers nobody has reviewed yet. ## A better wording that was worse The remaining mistakes had a pattern: the developer's next message was a follow-up question, and the model read it as `failed` or `worked` rather than `unclear`, with confidence between 0.43 and 0.57. The obvious fix was to say so in the spec. `failed` gained "A follow-up question (asking to explain, compare or recommend) is not a failure", and `unclear` gained "such as a follow-up question about the answer". Before shipping it, `hunch diff` compared the new wording with the old on the same 475 turns: * 108 of 475 turns changed answer. Most moved from `worked` to `unclear`. * On the 21 graded turns both versions shared, accuracy fell from 95.2% to 71.4%: 0 fixed, 5 broken. A line meant to rescue a handful of rows had moved more than a fifth of them. The old wording stayed. This is what `diff` is for: the change read well, and only the rows showed it was wrong. ## Grade from the text, not from memory When you review your own sessions, you remember what happened next and the model does not. Press `c` (needs more context) when only your memory decides a row; see [Review](/guides/review). ## What is still open * `claims` has not been reviewed on the maintainer's sessions. * The own-session grades came from the person who wrote the sessions. The review screen shows the model's answer, so the grades are not blind. * 24 random grades give a wide interval. About 30 more are needed before judging the next wording change.