Stacksift

Problem

Company websites mix real products, the things you can actually buy, with features, pricing tiers, AI add-ons and marketing names. If a sales or research database records "Dialpad AI" or "Zapier Tables" as a product, that mistake is expensive to clean up later.

So the question isn't "list every product-sounding name on the site". It's "could a customer buy this on its own, or is it part of something else?" That's the distinction my answer sheet is built to test.

Constraints

Being right matters more than being fast. Wrongly calling a known feature a product is worse than missing a product, because a miss gets flagged for a person to check.

The final decision step has to go through one company's candidates one at a time. Each decision sees the products already confirmed, so the same product under two different names gets caught. Running those decisions in parallel would mean none of them could see the others.

If the website can't be reached, the tool still has to return an answer based on search results alone, instead of crashing.

How it works

1. Search. It runs a general search for the company plus several targeted searches at the same time, using the Serper search API.

2. Read the site. It reads the homepage and the subpages most likely to mention products, up to a set limit on how many pages it fetches.

3. List candidates. A smaller, cheaper model (gpt-4.1-mini) pulls out up to 15 possible products.

4. Merge duplicates. It combines names that refer to the same thing. If this step fails, it keeps the list from step 3 as it was, instead of guessing at merges.

5. Decide. For each candidate it gathers supporting text from the pages, then a stronger model (gpt-4.1) decides product or not, one candidate at a time. Plain brand names like "Dialpad" go last, so specific product lines are decided first. If a decision fails, the answer becomes "uncertain, low confidence". It never quietly counts as a yes.

Every answer has to fit a fixed format that is checked automatically (Pydantic models with OpenAI's strict JSON Schema). Evidence links that point outside the company's own site are removed afterwards. The tool also refuses to read pages on a fixed list of outside sites, so it can't mistake a review site for the company's own website.

How I tested it

I wrote an answer sheet (data/eval_labels.json) with room for 32 companies. 20 are fully checked, and 12 are left for later. For each checked company it lists the real products and, separately, the names that look like products but aren't. Names are matched exactly first, then loosely, with a rule that a plain brand name can't count as a match for a specific product: "Dialpad" doesn't count as "Dialpad Support".

I ran one test on 1 September 2026, with search working and all 20 companies scored:

• Of the products it named, 77% were real products (precision 0.771).
• Of the real products, it found 87% (recall 0.869).
• The two combined give a single score of 0.795 (F1).
• In total it named 15 things that weren't real products and missed 18 that were.
• The whole run cost $1.10.

The answers, the tool's output and a script to re-score it are public at github.com/irfanalidv/stacksift-eval.

Automatically tuning the decision step made the score worse, so I undid it. The project README records why: the penalty for wrong answers was too small, the training used links instead of page text, and it lacked tricky "looks like a product but isn't" examples that have review-site listings.

Known failure modes

• Calling a feature a product

  Dialpad AI, HubSpot Breeze, Zapier Tables, Salesloft Rhythm and Loom AI all have marketing pages, and some have review-site listings, but you can't buy any of them on their own. My answer sheet lists them as "not products". The tool still sometimes calls them products when the page looks like a product page.

• An automatic tuning step made it worse

  I tried a tool (DSPy's MIPROv2) that automatically rewrites the instructions for the final decision step. The result scored worse, so I undid it. Three reasons: I trained it on links instead of the actual page text the model sees, it wasn't penalised enough for calling a feature a product, and the training examples didn't include features that also have review-site listings.

• Missing products at companies that sell several

  Companies with several products are where checking candidates one at a time helps most, and also where most misses still happen. Dialpad has three products in my answer sheet. The tool found two (Support and Sell), missed Meetings, and wrongly added "Dialpad AI Agents" and plain "Dialpad", both of which my answer sheet says are not products.

• Carrying on when search fails

  If the search service (Serper) rejects a request, the tool still treats that as "no search results" and carries on reading the website. The test now checks the search account has credit before it starts, and leaves out any company where search didn't run. In this test, all 20 companies were scored and none were left out. An earlier run that day, done with no search credit, is not in the published numbers.

What I would do differently

• Make the tool stop when search fails, the same way the test now does, instead of carrying on with no search results.
• Keep publishing the scoring pack (answer sheet, frozen output, re-scoring script, but not the tool itself) every time I re-test. My RAGNav library does the same with its published results file.
• Not use automatic tuning again until the training examples include tricky "not a product" cases with the full page text, not just links.
• When the merge step fails, it keeps everything, which is safe against crashes but lets duplicates through. I'd test that case on its own.