Engines

The engines a spec’s model can name, how an LLM is read through its token probabilities, and how to compare or escalate between them.

The spec’s model picks the engine. Every engine returns the same typed answers, so review, test and diff work the same whichever one answered.

model	Engine	Key
jev-1.13.0	TypeSafe’s Jev: a small model trained for typed judgments with calibrated probabilities	TYPESAFE_API_KEY
deepseek:<model id>	DeepSeek’s API, thinking disabled	DEEPSEEK_API_KEY
openrouter:<model id>[@provider]	Any model on OpenRouter that returns log-probabilities	OPENROUTER_API_KEY
distilled:<folder>	A local model trained by hunch distill on another engine’s answers: a frozen MiniLM encoder and a linear layer per question, on the CPU	none
<prefix>:<model>	An engine from an installed plugin package, registered under <prefix>	the plugin’s own

​

LLM engines

An LLM has no typed output, so hunch asks it for one token and reads that token’s probabilities.

• Prompt. The question and numbered options come first, so providers can cache that part across rows. The row follows as JSON. The model is asked for only the option number, yes/no, or the score level.
• Probabilities. hunch takes the top 20 log-probabilities of the first answer token and renormalises them over the valid answers.
• Temperature 1. At temperature 0 some APIs give every token but the top one a probability of zero, which would make every answer look certain.
• Providers. This needs log-probabilities with reasoning turned off. Some providers return none, or reason anyway, which hides the first answer token. Pin one that works with @provider:

hunch run . --model openrouter:<vendor>/<model>@<provider>

​

Compare engines: --model

The benchmark is every battery measured with every engine it has a receipt for, generated from those receipts.

--model overrides the spec’s engine for one command. Results go to their own table, <judgment>__<engine>, and each engine’s answers are cached under their own keys, so nothing is overwritten.

hunch run . --model deepseek:deepseek-flash --max-cost 0.50
hunch test . --model deepseek:deepseek-flash
hunch diff . --model deepseek:deepseek-flash      # row by row against the spec's own engine

On the same specs and rows, Jev and DeepSeek (deepseek-flash) measured as follows:

Task	Jev	DeepSeek
BANKING77 intent, 77 options	95.8%	92.3%	Jev better, p = 0.001
SWE-agent patches, would it pass (AUROC)	0.831	0.866	no significant difference
Claude Code turns, outcome / claims	90% / 88%	88% / 88%	no significant difference

DeepSeek cost about 1.5× Jev as billed. Both were well calibrated on BANKING77 (calibration error 0.043 for Jev, 0.032 for DeepSeek). Background in the cookbooks.

​

Engines from plugins

Any package can add an engine. Install one with hunch install hunch-engine-<name>, which puts it where hunch can see it, and hunch plugins lists what is installed. A package registers an object under the entry-point group hunch.engines, and the entry point’s name becomes the prefix a spec’s model uses:

pyproject.toml

[project.entry-points."hunch.engines"]
ollama = "hunch_engine_ollama:engine"      # model: ollama:<name>

The object needs one async method, called once per row with every question for it:

async def answer(self, model, state, questions) -> dict:
    # questions: {id: {"type", "instructions", "criteria"}}, as hunch sends them to Jev
    return {"answers": {id: answer, ...}, "cost": 0.0, "tokens": 0}

Name the package hunch-engine-<name>, so people find it the way they find llm-* or dbt-* packages. Answers take Jev’s shapes ({"type": "noul", "noul": p_yes}, a choice’s choice, confidence and probabilities, a score’s score, confidence, legend and probabilities), and hunch checks each one before storing it (the shape, that probabilities are in [0, 1], that a choice is one of its options), naming the question and what is wrong; a broken plugin stops the run with that message instead of being retried. Optional attributes: key, the environment variable it needs; worst_cost(model, state, questions), the most one call can be charged, so --max-cost holds (without it hunch treats the engine as free, and a capped run stops at the first call that reports a cost above what it reserved); adapter, a string in every answer’s cache key, to change when the engine’s prompt or parsing changes (without it, the key holds the plugin package’s name and version, so an upgrade never serves an older version’s answers); concurrency, calls in flight at once. A spec (or an escalate) that names a prefix no engine has fails lint and lists the engines installed. A plugin can’t take a built-in prefix (openrouter, deepseek, distilled): it is ignored, with a lint warning.

plugins/hunch-engine-ollama is the reference plugin, a package of its own: any model Ollama serves, asked with hunch’s own LLM prompt and read from its token probabilities, with smoke tests CI runs against this hunch. It also shows the case a plugin exists for: a specialised model that answers one question in its own format.

Bespoke-MiniCheck is a 7B model trained to say whether a document supports a claim. Asked the rag-answers battery’s question in hunch’s generic prompt, it scored AUROC 0.265 on a 40-answer sample: reliably backwards, since its “yes” means supported and it answered its own question, not the spec’s “does it say anything unsupported”. The plugin’s template asks it natively (Document: {passages} / Claim: {answer}) and turns its Yes into the spec’s no; the template’s name is part of the model, ollama:bespoke-minicheck#rag-unsupported, so its answers are cached and benchmarked apart. On all 900 answers, locally and for free:

Engine	AUROC	Accuracy
jev-1.13.0	0.941	83.8%
deepseek:deepseek-flash	0.907	77.3%
ollama:bespoke-minicheck#rag-unsupported	0.840	67.4%
ollama:qwen2.5:0.5b	0.434	21.2%

In its own format it ranks unsupported answers well, below Jev and DeepSeek on this data. The number said the generic prompt was wrong before anything relied on it; the benchmark keeps these side by side.

​

Escalate

A question can send its uncertain answers to a second engine. Answers below act are asked again on the escalate model, and the second answer replaces the first if it clears act itself (behind a distilled: model, always, since the teacher is the better model):

    act: 0.95
    escalate: {model: "deepseek:deepseek-flash"}

Both answers stay in the store. The table’s <question>_by column names the engine whose answer was used, and <question>_key points at that answer. See the spec reference.