Distill a decision into a local model

Train a small model on the answers your engine already gave, measure it like any engine, and let it answer what it is sure of while the engine answers the rest.

Every answer your engine gives is kept in the store, under its exact input. After a few thousand of them, the store holds something valuable: a labelled dataset for that exact question, in your data, with your reviews on top. hunch distill turns it into a small model that runs on your machine in about 10 milliseconds, with no API call.

A small model on its own is worse than the engine that taught it. It doesn’t have to be good at everything, though. It only has to know when it is sure. Put the engine behind it, and the student answers the rows it is sure of while the engine answers the rest. hunch measures both, so you choose how much to keep local with numbers.

Distill

The BANKING77 spec sorts bank customers’ messages into 77 intents. Jev had answered 3,000 unlabelled messages, kept in the store. Distilling asks nothing:

cd prototype/examples/banking77
hunch distill intent.yml --source banking77_unlabeled.csv

intent: distilled to distilled/intent@55a07d9f
  intent: 3000 answers, 77 classes; rarest: 'contactless_not_working' (7)  ← few examples: measure what it misses (hunch test --model) before trusting it
measure it (free, local): hunch test intent.yml --model distilled:distilled/intent@55a07d9f
use it: model: distilled:distilled/intent@55a07d9f, with escalate: {model: jev-1.13.0} and act on each question, so answers it isn't sure of go to jev-1.13.0

Each row’s state goes through a small frozen encoder (MiniLM, on your CPU), and each question gets one linear layer, trained on your gold where a row has it and on the engine’s answer everywhere else. One more number, a temperature, is fitted on answers each part of the data got from a model that never saw it, so the student’s confidence means what it says: of the answers it gives at 0.9, about nine in ten are right. If those answers are almost never wrong, there is nothing to fit it on, and the confidence is left as it is. The rarest answer is reported because a student learns little from a handful of examples. The folder, 194 KB here, is the model; commit it with the spec.

It needs the distill extra: uv add "hunch-ai[distill]", which brings fastembed and its runtime (onnxruntime, tokenizers and about 20 other packages). The encoder downloads once, about 90 MB, to ~/.cache/hunch/encoders (or FASTEMBED_CACHE_PATH).

Measure it like any engine

A distilled model is an engine, distilled:<folder>, so every command measures it. On the 385 held-out messages, which it never saw:

hunch test intent.yml --source banking77_holdout.csv --model distilled:distilled/intent@55a07d9f --max-cost 0

intent (choice, 385 rows)
  PASS accuracy 91.2% (raw source gold: 85.7%) (min 85%)
  PASS calibration error 0.033 (raw source gold: 0.049) (max 0.05)
       dial   automated   wrong among automated
       0.70       86.8%                    3.9%
       0.80       81.0%                    1.6%
       0.90       73.2%                    1.1%  ← act
       0.95       66.2%                    0.8%

On its own it is right 91.2% of the time, where Jev is right 97.9%. It knows which answers those are: at 0.9 and above it answers three messages in four, and gets about one in a hundred wrong. distill holds back a fifth of any gold rows it could have learned from and records what it trained on, so test and diff grade a distilled model only on rows it never saw. (Never saw means never saw that exact text: 3 of these 385 messages differ from a training message only in case or punctuation.)

Put the engine behind it

intent_distilled.yml is intent.yml with the student in front and Jev behind it:

intent_distilled.yml

model: distilled:distilled/intent@55a07d9f
questions:
  intent:
    # type, instructions, criteria as in intent.yml
    act: 0.90
    escalate: {model: jev-1.13.0}    # below 0.90, the teacher answers

hunch test intent_distilled.yml --source banking77_holdout.csv --max-cost 0

intent (choice, 385 rows)
  PASS estimated accuracy 95.5% (95% CI 88.3%–97.6%) from reviews of 60/343 agreeing rows, 42/42 disagreeing rows (min 85%)
  PASS calibration error 0.032 (raw source gold: 0.054) (max 0.05)
  PASS accuracy among auto-acted 99.1% on 88% of rows at act=0.9 (min 97%)
  PASS order stability: 0/300 answers flip (0.0%, 0 within noise band), mean |Δp| of original answer 0.000 (max flip rate 5%)
  escalated to jev-1.13.0: 103/385 rows use its answer, 55 of them confident enough to act; right on 97/103 with gold

Jev alone, measured the same way, is estimated at 95.8% (95% CI 88.7%–97.9%); on plain accuracy the two are 97.9% and 97.7%, one message apart. Nearly three messages in four (282 of 385) were answered on this machine. Behind a distilled model, an escalated answer always replaces the student’s: the teacher is the better model, so every doubt goes to it.

The share that stays local grows with the answers you keep. Students trained on more of Jev’s answers, measured on the same holdout with a script that calls hunch’s own training, kept more:

Jev’s answers learned from	answered locally at 0.9	combined plain accuracy
770	59%	97.7%
1,770	71%	97.9%
3,770	72%	97.7%

Every run and every review adds to the store, so distilling again later keeps more of the work local.

When not to use it

A student learns an answer from examples of it. When one answer is rare and missing it is costly, a few thousand rows aren’t enough. The command guard is that case: 2% of shell commands need a person, so its student saw only 11 destructive commands. Measured by a script on commands the panel had labelled and the student never saw, the student with Jev behind it (act 0.9) let through 2 of 6 destructive commands and 1 of 12 that send data out, where Jev alone let through none. distill flags the rare answers; hunch test --model distilled:… shows the misses on the dial’s “act on no” side. For a guard like that, keep the engine until the store holds many more of the rare answer.

A distilled model answers only the questions it was trained on, word for word. Change a question and it refuses until you distill again, and hunch diff against the old engine shows what the new student changed.

Speed and cost

Inside a running app, a new answer takes about 11 ms: the encoder and the linear layer, on the CPU. A fresh process pays about 0.3 s to load the encoder first, so a per-command hook gains no speed, only cost and network. Distilled answers cost nothing and are stored like any other, so a row seen before comes from the store.