Match two product catalogues

Two shops list the same products under different names. Teach a program which ones match, and to say when it isn’t sure.

Two online shops, Abt and Buy, sell about a thousand products each. Which products are the same?

Joining two lists of the same things is everywhere: catalogues after an acquisition, suppliers in two systems, customers who signed up twice. Rules get you most of the way. The rest is a judgment, and that’s the part hunch is for.

This benchmark has known matches, so it lets us count errors. The question was improved after inspecting these same pairs; its final accuracy is a result on this dataset, not an estimate of how it will do on a new catalogue.

​

Try it first

Pick the one that’s the same product. Then see what simple rules chose, and what hunch said.

Rules compare characters, so a case and a cable one digit apart look alike. The question compares meaning, and says how sure it is. When nothing fits, it says so.

​

1. Narrow the pairs with rules

Put every Abt product next to every Buy product and you get 1,081 × 1,092 pairs. Almost all of them are nonsense: a turntable next to photo paper. There’s no point asking anyone about those, so two cheap clues, both plain SQL, throw them out first.

Clue 1: a shared word. Split each name into words and keep a pair if the Abt name shares at least one word with the Buy name or manufacturer. This cheap filter drops 85% of all pairs. It can also drop real matches whose listings use entirely different words.

Clue 2: a shared model code. Many names carry a model code, a word of at least four characters that mixes letters and digits. The query drops hyphens, slashes and dots, then treats equal codes or codes of at least five characters where one contains the other as shared:

Listing	Its model code
Sony Turntable - PSLX350H	PSLX350H
Sony PS-LX350H Belt-Drive Turntable	PSLX350H
Sony PSLX300USB USB Record Turntable	PSLX300USB

Keep the five best. For each Abt product, candidates are sorted by shared code, then the fraction of distinct words they share, then Buy id to break ties. The top five stay. Here is that for one product, the turntable. Press play.

The true match ranks first because it shares the code, even though the sibling turntable has a higher word-overlap score. Five per product gives 5,396 pairs, and 1,061 of the 1,097 true matches are still among them. The remaining 36 cannot be recovered by the model because they never reach it.

In candidates.py the two clues are one query. shared_words, shares_code and word_overlap are small SQL functions defined at the top of that file, and they do exactly what their names say:

-- every pair of products that shares at least one word
with b as (
    select id,
           name || ' ' || manufacturer as words_text,
           name || ' ' || description  as codes_text
    from buy
)
select abt.id as abt_id,
       b.id   as buy_id,
       shares_code(abt.name, b.codes_text)  as code,
       word_overlap(abt.name, b.words_text) as overlap
from abt, b
where shared_words(abt.name, b.words_text) > 0

Then each product keeps its five best:

row_number() over (partition by abt_id
                   order by code desc, overlap desc) <= 5

​

2. Write the question

One file says what to ask about each pair, and what the model may read:

same_product.yml

judgment: same_product
model: jev-1.13.0
source: py(candidates.py:pairs)
key: id
state: [abt_name, abt_description, abt_price,
        buy_name, buy_manufacturer, buy_description, buy_price]
questions:
  same:
    type: noul          # yes or no, with how sure
    instructions: >-
      Two shops list a product: `abt_name` (with
      `abt_description` and `abt_price`) and `buy_name` (with
      `buy_manufacturer`, `buy_description` and `buy_price`).
      Are they the same product: the same maker and the same
      model? A different model number, size or capacity is a
      different product, and so is an accessory made for it. The
      two shops word names differently, and a price may be
      missing. A shop may also add its own letters to the end of
      the same model's code (AB123 in one shop, AB123XY in the
      other): when the rest of the code and the product agree,
      it is the same product. But when both listings name a
      colour and the colours differ, they are different
      products.
    gold: gold_same     # the known answer
    act: 0.9            # below this, a person decides

​

3. Run it

hunch run prototype/examples/product_matching/same_product.yml \
  --max-cost 0.25

All 5,396 pairs take about 90 seconds and $0.14. Every answer is kept, so running it again is free.

​

4. Keep one match per product

A product can have two candidates that both look right. Back in SQL, where the answer is just a column, keep the likeliest pair on both sides:

-- simplified from match.py, which also breaks ties
select * from (
    select * from decided
    where label = 'yes'
    qualify row_number() over (partition by abt_id
                               order by p_yes desc) = 1
)
qualify row_number() over (partition by buy_id
                           order by p_yes desc) = 1

match.py runs all four steps.

​

The result

And the 682 matches it was sure of were all right.

​

Changing the question safely

The question above is the third version. The second taught a lesson.

Version one said nothing about shops adding letters to a code, so PM1327BK and PM1327 came back as different products. The obvious fix: add a sentence saying such letters, “for a colour, a finish or a region”, don’t make a different product.

Before shipping it, hunch diff asked the new wording of every pair:

gold accuracy on the 5286 shared rows with gold 98.6% → 98.3%
  (✓ 23 fixed, ✗ 40 broken, 0 wrong both times, 5 without gold)

It fixed 23 pairs and broke 40. A silver and a gold camera of the same model are two products, and “a colour” had just told the model they were one. The shipped wording draws the line at colours: 35 fixed, 4 broken.

The words of a question are code. Test an edit on every row before it ships.

​

Use it on your data

Point the read_csv lines in candidates.py at your two files, keep the question, review a few dozen pairs, and hunch test tells you how often it’s right on your products.

​

How it was measured

The data

Abt-Buy, a benchmark from the University of Leipzig (Creative Commons): 1,081 products from Abt, 1,092 from Buy, 1,097 known matches. Comparing all of them is 1.18 million pairs; candidates.py keeps up to five per Abt product, 5,396 pairs, which hold 1,061 of the 1,097 matches. The other 36 are lost during candidate selection, either at the shared-word filter or the top-five ranking. Asking all 5,396 took about 90 seconds and $0.14.

The results, in full

Against the benchmark’s answer key: each product’s best candidate by the rules alone, 955 of 1,081 right; the rules, the question and the query, 971 of 980 right (precision 99.1%, recall 88.5%); keeping only matches above the bar, 682 of 682. 4,797 of the 5,396 pairs clear act: 0.9. For 51 products the rules’ two best candidates tie and the lower id wins. Since 16 products have more than one true match, one match per product can reach at most 1,044 of the 1,097.

Checking the answer key

The first version of the question disagreed with the benchmark’s answer key on 180 pairs, and its most confident disagreements looked like the key’s mistakes (a 500GB drive matched to a 1TB one). A blind panel of three AI reviewers (one Claude Opus, two Claude Sonnet, one reading in reverse order) judged those 180 and a random 100, seeing what the model saw and neither the model’s answer nor the key’s. On the 180, the key was right 69 times and the model 22; 81 could not be decided from the listings, and on 8 the reviewers split. On the random 100, the panel agreed with both on 96. Two later rounds covered the new wordings’ fresh disagreements (11 pairs, then 4). The reviewers’ prompt is in prototype/review_panel/product_matching/PROMPT.md; it told them a trailing colour or region suffix can be formatting, a hint the first version of the question lacked.

Accuracy, with its interval

With the key corrected by the panel, hunch test estimates the question is right on 99.2% of pairs (95% CI 95.4–99.2%), from 100 random reviews and every disagreement. Of the 4,797 pairs above the bar, 2 are wrong against that gold, both a confident “no” for the same product under a shop’s own code (a Maytag microwave and a Sanus wall mount). Against the uncorrected key 15 are, and the panel judged 12 of those the key’s mistake.

The question was tuned on the pairs it is scored on

The code-suffix pattern was found in the panel’s verdicts, and each version was judged by diff against the same gold, so the final numbers are somewhat optimistic for listings like these that it hasn’t seen. The first two versions are kept as first/same_product.yml and tried/same_product.yml, so both diffs can be run again for free.

What did not work

A small local model trained with hunch distill, which would make matching free, separated matches from non-matches barely better than a coin (AUROC 0.576, where the question scores 1.000). It reads both listings as one piece of text and loses whether EX85 and EX81 are the same code. And the panel is not people: on the hard pairs, one reviewer said “same” on 60 pairs the other two called undecidable. For your own catalogues, a person reviewing the pairs below the bar is the better answer key.