same? {c[1] >= 0.5 ? "yes" : "no"}
= 0.5 ? teal : "rgba(128,128,128,0.55)"
}} />
{c[1].toFixed(2)}
}
;
};
return
Product {i + 1} of {puzzles.length}. Which of these is the same product?
{z.product}
{z.candidates.map((c, k) => row(c[0], k))}
{row("None of these", -1)}
{shown &&
{right ? "Right." : "Not quite."}{" "}
The rules picked the first one{match === 0 ? ", and that's right" : ", and that's wrong"}.{" "}
The question said {hunchPick < 0 ? "none of them" : `yes to “${z.candidates[hunchPick][0]}”`}
{hunchPick === match ? ", and that's right." : "."} {z.why}
}
;
};
export const PairFunnel = ({steps}) => {
const max = steps[0][0];
return
{steps.map(([n, what]) =>
{what}{n.toLocaleString("en")}
)}
;
};
Two online shops, Abt and Buy, sell about a thousand products each. Which products are the same?
Joining two lists of the same things is everywhere: catalogues after an acquisition, suppliers in two systems, customers who signed up twice. Rules get you most of the way. The rest is a judgment, and that's the part hunch is for.
## Try it first
Pick the one that's the same product. Then see what simple rules chose, and what hunch said.
Rules compare characters, so a case and a cable one digit apart look alike. The question compares meaning, and says how sure it is. When nothing fits, it says so.
## 1. Narrow the pairs with rules
Put every Abt product next to every Buy product and you get 1,081 × 1,092 pairs. Almost all of them are nonsense: a turntable next to photo paper. There's no point asking anyone about those, so two cheap clues, both plain SQL, throw them out first.
**Clue 1: a shared word.** Split each name into words. If two listings have no word in common, they are not the same product. That alone drops 85% of the pairs.
**Clue 2: a shared model code.** Most names carry a model code, a word that mixes letters and digits. Shops write codes differently, so hyphens, slashes and dots are dropped before comparing:
| Listing | Its model code |
| ------------------------------------ | -------------- |
| Sony Turntable - PSLX350H | `PSLX350H` |
| Sony PS-LX350H Belt-Drive Turntable | `PSLX350H` |
| Sony PSLX300USB USB Record Turntable | `PSLX300USB` |
**Keep the five best.** For each product, its candidates are sorted: a shared code first, then the most words in common, and the top five stay. Here is that for one product, the turntable. Press play.
The true match ranks first because it shares the code, even though the sibling turntable shares more of its words (2 of 6, against 2 of 7). Five per product gives 5,396 pairs, and 1,061 of the 1,097 true matches are still among them.
In [`candidates.py`](https://github.com/oneryalcin/hunch/blob/main/prototype/examples/product_matching/candidates.py) the two clues are one query. `shared_words`, `shares_code` and `word_overlap` are small SQL functions defined at the top of that file, and they do exactly what their names say:
```sql theme={null}
-- every pair of products that shares at least one word
with b as (
select id,
name || ' ' || manufacturer as words_text,
name || ' ' || description as codes_text
from buy
)
select abt.id as abt_id,
b.id as buy_id,
shares_code(abt.name, b.codes_text) as code,
word_overlap(abt.name, b.words_text) as overlap
from abt, b
where shared_words(abt.name, b.words_text) > 0
```
Then each product keeps its five best:
```sql theme={null}
row_number() over (partition by abt_id
order by code desc, overlap desc) <= 5
```
## 2. Write the question
One file says what to ask about each pair, and what the model may read:
```yaml same_product.yml theme={null}
judgment: same_product
model: jev-1.13.0
source: py(candidates.py:pairs)
key: id
state: [abt_name, abt_description, abt_price,
buy_name, buy_manufacturer, buy_description, buy_price]
questions:
same:
type: noul # yes or no, with how sure
instructions: >-
Two shops list a product: `abt_name` (with
`abt_description` and `abt_price`) and `buy_name` (with
`buy_manufacturer`, `buy_description` and `buy_price`).
Are they the same product: the same maker and the same
model? A different model number, size or capacity is a
different product, and so is an accessory made for it. The
two shops word names differently, and a price may be
missing. A shop may also add its own letters to the end of
the same model's code (AB123 in one shop, AB123XY in the
other): when the rest of the code and the product agree,
it is the same product. But when both listings name a
colour and the colours differ, they are different
products.
gold: gold_same # the known answer
act: 0.9 # below this, a person decides
```
## 3. Run it
```sh theme={null}
hunch run prototype/examples/product_matching/same_product.yml \
--max-cost 0.25
```
All 5,396 pairs take about 90 seconds and \$0.14. Every answer is kept, so running it again is free.
## 4. Keep one match per product
A product can have two candidates that both look right. Back in SQL, where the answer is just a column, keep the likeliest pair on both sides:
```sql theme={null}
-- simplified from match.py, which also breaks ties
select * from (
select * from decided
where label = 'yes'
qualify row_number() over (partition by abt_id
order by p_yes desc) = 1
)
qualify row_number() over (partition by buy_id
order by p_yes desc) = 1
```
[`match.py`](https://github.com/oneryalcin/hunch/blob/main/prototype/examples/product_matching/match.py) runs all four steps.
## The result
And the 682 matches it was sure of were all right.
## Changing the question safely
The question above is the third version. The second taught a lesson.
Version one said nothing about shops adding letters to a code, so `PM1327BK` and `PM1327` came back as different products. The obvious fix: add a sentence saying such letters, "for a colour, a finish or a region", don't make a different product.
Before shipping it, `hunch diff` asked the new wording of every pair:
```text theme={null}
gold accuracy on the 5286 shared rows with gold 98.6% → 98.3%
(✓ 23 fixed, ✗ 40 broken, 0 wrong both times, 5 without gold)
```
It fixed 23 pairs and broke 40. A silver and a gold camera of the same model are two products, and "a colour" had just told the model they were one. The shipped wording draws the line at colours: 35 fixed, 4 broken.
The words of a question are code. Test an edit on every row before it ships.
## Use it on your data
Point the `read_csv` lines in `candidates.py` at your two files, keep the question, [review](/guides/review) a few dozen pairs, and `hunch test` tells you how often it's right on your products.
## How it was measured
[Abt-Buy](https://dbs.uni-leipzig.de/research/projects/benchmark-datasets-for-entity-resolution), a benchmark from the University of Leipzig (Creative Commons): 1,081 products from Abt, 1,092 from Buy, 1,097 known matches. Comparing all of them is 1.18 million pairs; `candidates.py` keeps five per Abt product, 5,396 pairs, which hold 1,061 of the 1,097 matches. The 36 lost share a few words with their match, but five other listings outrank them. Asking all 5,396 took about 90 seconds and \$0.14.
Against the benchmark's answer key: each product's best candidate by the rules alone, 955 of 1,081 right; the rules, the question and the query, 971 of 980 right (precision 99.1%, recall 88.5%); keeping only matches above the bar, 682 of 682. 4,797 of the 5,396 pairs clear `act: 0.9`. For 51 products the rules' two best candidates tie and the lower id wins. Since 16 products have more than one true match, one match per product can reach at most 1,044 of the 1,097.
The first version of the question disagreed with the benchmark's answer key on 180 pairs, and its most confident disagreements looked like the key's mistakes (a 500GB drive matched to a 1TB one). A blind panel of three AI reviewers (one Claude Opus, two Claude Sonnet, one reading in reverse order) judged those 180 and a random 100, seeing what the model saw and neither the model's answer nor the key's. On the 180, the key was right 69 times and the model 22; 81 could not be decided from the listings, and on 8 the reviewers split. On the random 100, the panel agreed with both on 96. Two later rounds covered the new wordings' fresh disagreements (11 pairs, then 4). The reviewers' prompt is in `prototype/review_panel/product_matching/PROMPT.md`; it told them a trailing colour or region suffix can be formatting, a hint the first version of the question lacked.
With the key corrected by the panel, `hunch test` estimates the question is right on 99.2% of pairs (95% CI 95.4–99.2%), from 100 random reviews and every disagreement. Of the 4,797 pairs above the bar, 2 are wrong against that gold, both a confident "no" for the same product under a shop's own code (a Maytag microwave and a Sanus wall mount). Against the uncorrected key 15 are, and the panel judged 12 of those the key's mistake.
The code-suffix pattern was found in the panel's verdicts, and each version was judged by `diff` against the same gold, so the final numbers are somewhat optimistic for listings like these that it hasn't seen. The first two versions are kept as `first/same_product.yml` and `tried/same_product.yml`, so both diffs can be run again for free.
A small local model trained with [`hunch distill`](/guides/distill), which would make matching free, separated matches from non-matches barely better than a coin (AUROC 0.576, where the question scores 1.000). It reads both listings as one piece of text and loses whether `EX85` and `EX81` are the same code. And the panel is not people: on the hard pairs, one reviewer said "same" on 60 pairs the other two called undecidable. For your own catalogues, a person reviewing the pairs below the bar is the better answer key.