Stacksift

Problem

B2B product pages mix independently purchasable SKUs with features, tiers, AI layers, and marketing labels. A downstream record that treats “Dialpad AI” or “Zapier Tables” as a product is wrong in a way that is expensive to unwind.

The judgement is not “extract every noun phrase.” It is “would a buyer write a PO for this, or is it a bundled capability?” That distinction is what the golden set is for.

Constraints

Accuracy over speed. A false positive that promotes a known feature is worse than a miss that queues for review.

Pass 3 must run sequentially per domain. Each verdict receives other_confirmed_products from earlier in the loop so the same SKU under two names can be rejected. Parallelising that loop would give every call an empty list.

When the site is unreachable, the pipeline still has to return something: a search-only fallback, not a crash.

Architecture

Stage 1 gathers Serper intelligence (overview plus concurrent keyword queries). Stage 2 crawls the homepage and signal-bucketed subpages under a fetch budget. Stage 3 (Pass 1, gpt-4.1-mini) extracts candidates under a cap of 15. Stage 4 (Pass 2) merges duplicates; on parse failure it passthrough-copies Pass 1 rather than inventing a merge.

Stage 5 gathers per-candidate evidence concurrently, then runs Pass 3 (gpt-4.1 / DSPy ChainOfThought) in a for-loop. Bare-brand names are sorted last so product-line verdicts land first. Pass 3 parse or call failure materialises verdict=uncertain, confidence=low — it does not fail open.

Outputs are Pydantic models with OpenAI strict JSON Schema. Off-domain evidence URLs are stripped after the model, not inside the prompt. A scrape-guard blocks a hardcoded list of third-party hosts so the crawler cannot wander onto review platforms as if they were the target site.

Reliability / eval

The labelled set lives in data/eval_labels.json: 32 domain slots, 20 verified, 12 left as TBD for later labelling. Each verified row has true_products and not_products. Scoring is exact-then-fuzzy name matching with a bare-brand guard so “Dialpad” cannot consume “Dialpad Support.”

Frozen 2026-09-01 (one run, search live, 20/20 scored, nothing excluded): macro precision 0.771, recall 0.869, F1 0.795. Fifteen false positives, eighteen false negatives, $1.10 metered. Predictions and rescore: github.com/irfanalidv/stacksift-eval.

A DSPy MIPROv2 compile of Pass 3 made the metric worse; I reverted it. Causes recorded in the README: FP penalty too weak, training on URL refs instead of evidence snippets, missing hard negatives with review-site listings.

Known failure modes

• Feature-as-product

  Dialpad AI, HubSpot Breeze, Zapier Tables, Salesloft Rhythm, Loom AI: marketing pages and sometimes review listings exist, but they are not independently purchasable. The golden set encodes that as not_products. The model still promotes them when evidence looks like a SKU page.

• Optimizer regression

  A MIPROv2 compile of Pass 3 made the metric worse. Training used URL references instead of real evidence snippets, the false-positive penalty in verdict_metric was −0.25 instead of −0.5, and the train set lacked not_product examples that also have G2/Capterra listings. I reverted the compiled module.

• Recall on multi-SKU sites

  Multi-product companies are where sequential Pass 3 earns its keep — each verdict sees already-confirmed products — and where misses still concentrate. Dialpad on the freeze: P 0.500, R 0.667, F1 0.571. Support and Sell matched; Meetings missed; predicted “Dialpad AI Agents” and bare “Dialpad” are labelled not_products.

• Search-provider fail-soft

  The analyzer still logs Serper 4xx as empty search and crawls anyway. The eval harness now probes GET /account before domain 1 and excludes any row that is not status=scored. This freeze: 20 scored, 0 excluded. An earlier same-day run with empty credits is not in the published numbers.

What I would do differently

• Fail the analyzer the same way the eval now fails: Serper 4xx should abort the request, not continue as empty Stage 1.
• The scoring pack is public (labels, frozen predictions, rescore, no analyzer), same shape as RAGNav committing squad_results.txt. I would keep shipping that pack with every freeze.
• I would not run MIPROv2 again until the train set has hard negatives with full evidence snippets, not URLs.
• Pass 2 passthrough on parse failure is conservative for crashes and generous for duplicates. I would isolate that as its own eval slice.