> ## Documentation Index > Fetch the complete documentation index at: https://fuguai.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Engines > The engines a spec's model can name, how an LLM is read through its token probabilities, and how to compare or escalate between them. The spec's `model` picks the engine. Every engine returns the same typed answers, so review, test and diff work the same whichever one answered. | `model` | Engine | Key | | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | `jev-1.13.0` | TypeSafe's Jev: a small model trained for typed judgments with calibrated probabilities | `TYPESAFE_API_KEY` | | `deepseek:` | DeepSeek's API, thinking disabled | `DEEPSEEK_API_KEY` | | `openrouter:[@provider]` | Any model on OpenRouter that returns log-probabilities | `OPENROUTER_API_KEY` | | `distilled:` | A local model trained by `hunch distill` on another engine's answers: a frozen MiniLM encoder and a linear layer per question, on the CPU | none | | `:` | An engine from an installed plugin package, registered under `` | the plugin's own | ## LLM engines An LLM has no typed output, so hunch asks it for one token and reads that token's probabilities. * **Prompt.** The question and numbered options come first, so providers can cache that part across rows. The row follows as JSON. The model is asked for only the option number, `yes`/`no`, or the score level. * **Probabilities.** hunch takes the top 20 log-probabilities of the first answer token and renormalises them over the valid answers. * **Temperature 1.** At temperature 0 some APIs give every token but the top one a probability of zero, which would make every answer look certain. * **Providers.** This needs log-probabilities with reasoning turned off. Some providers return none, or reason anyway, which hides the first answer token. Pin one that works with `@provider`: ```sh theme={null} hunch run . --model openrouter:/@ ``` ## Compare engines: `--model` The [benchmark](/reference/benchmark) is every battery measured with every engine it has a receipt for, generated from those receipts. `--model` overrides the spec's engine for one command. Results go to their own table, `__`, and each engine's answers are cached under their own keys, so nothing is overwritten. ```sh theme={null} hunch run . --model deepseek:deepseek-flash --max-cost 0.50 hunch test . --model deepseek:deepseek-flash hunch diff . --model deepseek:deepseek-flash # row by row against the spec's own engine ``` Measured so far, Jev against DeepSeek (`deepseek-flash`) on the same specs and rows: | Task | Jev | DeepSeek | | | ---------------------------------------- | --------- | --------- | ------------------------- | | BANKING77 intent, 77 options | 95.8% | 92.3% | Jev better, p = 0.001 | | SWE-agent patches, would it pass (AUROC) | 0.831 | 0.866 | no significant difference | | Claude Code turns, outcome / claims | 90% / 88% | 88% / 88% | no significant difference | DeepSeek cost about 1.5× Jev as billed. Both were well calibrated on BANKING77 (calibration error 0.043 for Jev, 0.032 for DeepSeek). Background in the [cookbooks](/cookbooks). ## Engines from plugins Any package can add an engine. Install one with `hunch install hunch-engine-`, which puts it where hunch can see it, and `hunch plugins` lists what is installed. A package registers an object under the entry-point group `hunch.engines`, and the entry point's name becomes the prefix a spec's `model` uses: ```toml pyproject.toml theme={null} [project.entry-points."hunch.engines"] ollama = "hunch_engine_ollama:engine" # model: ollama: ``` The object needs one async method, called once per row with every question for it: ```python theme={null} async def answer(self, model, state, questions) -> dict: # questions: {id: {"type", "instructions", "criteria"}}, as hunch sends them to Jev return {"answers": {id: answer, ...}, "cost": 0.0, "tokens": 0} ``` Name the package `hunch-engine-`, so people find it the way they find `llm-*` or `dbt-*` packages. Answers take Jev's shapes (`{"type": "noul", "noul": p_yes}`, a choice's `choice`, `confidence` and `probabilities`, a score's `score`, `confidence`, `legend` and `probabilities`), and hunch checks each one before storing it (the shape, that probabilities are in \[0, 1], that a choice is one of its options), naming the question and what is wrong; a broken plugin stops the run with that message instead of being retried. Optional attributes: `key`, the environment variable it needs; `worst_cost(model, state, questions)`, the most one call can be charged, so `--max-cost` holds (without it hunch treats the engine as free, and a capped run stops at the first call that reports a cost above what it reserved); `adapter`, a string in every answer's cache key, to change when the engine's prompt or parsing changes (without it, the key holds the plugin package's name and version, so an upgrade never serves an older version's answers); `concurrency`, calls in flight at once. A spec (or an `escalate`) that names a prefix no engine has fails lint and lists the engines installed. A plugin can't take a built-in prefix (`openrouter`, `deepseek`, `distilled`): it is ignored, with a lint warning. `plugins/hunch-engine-ollama` is the reference plugin, a package of its own: any model [Ollama](https://ollama.com) serves, asked with hunch's own LLM prompt and read from its token probabilities, with smoke tests CI runs against this hunch. It also shows the case a plugin exists for: a specialised model that answers one question in its own format. Bespoke-MiniCheck is a 7B model trained to say whether a document supports a claim. Asked the [rag-answers](/cookbooks/rag-answers) battery's question in hunch's generic prompt, it scored AUROC 0.265 on a 40-answer sample: reliably backwards, since its "yes" means supported and it answered its own question, not the spec's "does it say anything unsupported". The plugin's template asks it natively (`Document: {passages}` / `Claim: {answer}`) and turns its Yes into the spec's no; the template's name is part of the model, `ollama:bespoke-minicheck#rag-unsupported`, so its answers are cached and benchmarked apart. On all 900 answers, locally and for free: | Engine | AUROC | Accuracy | | ------------------------------------------ | ----- | -------- | | `jev-1.13.0` | 0.941 | 83.8% | | `deepseek:deepseek-flash` | 0.907 | 77.3% | | `ollama:bespoke-minicheck#rag-unsupported` | 0.840 | 67.4% | | `ollama:qwen2.5:0.5b` | 0.434 | 21.2% | In its own format it ranks unsupported answers well, below Jev and DeepSeek on this data. The number said the generic prompt was wrong before anything relied on it; the [benchmark](/reference/benchmark) keeps these side by side. ## Escalate A question can send its uncertain answers to a second engine. Answers below `act` are asked again on the `escalate` model, and the second answer replaces the first if it clears `act` itself (behind a `distilled:` model, always, since the teacher is the better model): ```yaml theme={null} act: 0.95 escalate: {model: "deepseek:deepseek-flash"} ``` Both answers stay in the store. The table's `_by` column names the engine whose answer was used, and `_key` points at that answer. See the [spec reference](/reference/spec).