Irfan Ali
$cd /writing
writing/evaluating-stacksift-verdict-pipeline

Eval / postmortem

Evaluating the Stacksift verdict pipeline

Frozen eval: macro P/R/F1 0.771 / 0.869 / 0.795 on 20 labelled domains. Dialpad is 0.571. The rest is the misses — including an optimizer that made the metric worse.

One freeze, 2026-09-01. Search was live. Twenty verified domains, all status=scored, none excluded. Macro precision 0.771, recall 0.869, F1 0.795. Fifteen false positives, eighteen false negatives. Metered cost $1.10. Predictions and a no-keys rescore live in github.com/irfanalidv/stacksift-eval. That file is the scoreboard. This page is the misses.

The freeze

Extract: gpt-4.1-mini. Verdict: gpt-4.1, uncompiled DSPy. Stacksift commit a8b6cb9. One run. Not repeated to improve the numbers.

Dialpad, three labelled SKUs: precision 0.500, recall 0.667, F1 0.571. Support and Sell matched. Meetings missed. Predicted “Dialpad AI Agents” and bare “Dialpad” are both labelled not_products and tagged matches_known_not_product. Sequential Pass 3 did not save Meetings, and it still promoted two traps I had written down.

Dialpad's F1 moved across runs (0.333 → 0.667 → 0.571), so the per-domain metric is noisy at n=1.

Worst F1 is 0. demandbase.com predicted the bare brand against three labelled SKUs. c3.ai predicted a different catalogue than the labels. Those zeros are in the macro. They are not incidents. They are the pipeline on those sites.

Other feature-as-product hits on this run: Zapier Tables, Algolia AI Browse. HubSpot missed Content Hub and Operations Hub, and invented Revenue Hub.

How scoring works

scripts/eval.py runs the five-stage pipeline against verified rows only. Names are matched exact-canonical first, then fuzzy with whole-word containment. A one-token prediction must not consume a multi-token label that merely shares the brand: predicted “Dialpad” does not match labelled “Dialpad Support.”

Precision is matched predictions over predictions. Recall is matched labels over true_products. False positives that also match a labelled not_product are tagged matches_known_not_product. Those are worse than an invented name: the model promoted a feature I had already written down as a trap.

Only status=scored enters the macro. Zero successful Serper calls is search_unavailable and is listed separately, not averaged in as 0/0/0. This freeze: 20 scored, 0 excluded. Empty prediction against a non-empty label is still a total miss when search actually ran.

How the golden set is built

data/eval_labels.json is a list of 32 domain slots. Twenty are marked verified: true with timestamp 2026-04-18. Twelve are left unverified (TBD) and are skipped when the harness computes metrics.

Each verified row has two lists, not one. true_products is what a buyer can write a PO for. not_products is the trap: names that appear on the marketing site, and often on G2 or Capterra, but are features, tiers, AI layers, or the bare brand. The second list is the point of the set. Without it, a pipeline that extracts every noun phrase looks accurate.

Examples, copied from the labels:

  • Dialpad: Support, Sell, Meetings are products. Dialpad (bare brand), Dialpad AI, DialpadGPT, and a cluster of “enterprise call center” paraphrases are not.
  • HubSpot: the seven Hubs are products. Breeze / Breeze AI, Free CRM, and “Smart CRM” are not. The notes flag that HubSpot renames Hubs (CMS Hub → Content Hub in 2024), so this row has to be re-checked.
  • Zapier: one product. Tables, Interfaces, Canvas, Chatbots are modules on a single pricing page.
  • Salesloft: one product. Rhythm is an AI workflow layer, not a SKU — and it has review-site listings, which is why it belongs in the train set as a hard negative.
  • Loom: Loom is the product. Loom AI is a bundled add-on.

The labelling rule I used: if it is not independently purchasable, it is not_products, even when a review site has a page for it. That is the failure mode I actually care about — feature-as-product — because it corrupts a downstream record in a way that is expensive to unwind.

The two product-side failures

Feature-as-product is still the recurring miss. Marketing pages describe Dialpad AI Agents, Zapier Tables, Algolia AI Browse, HubSpot Breeze, Salesloft Rhythm, and Loom AI as if they were products. Sometimes a review listing exists. Pass 3 still promotes them when the evidence looks like a SKU page. The golden set encodes the opposite. The freeze is the gap, measured.

Then the optimizer. A MIPROv2 compile of Pass 3 made the metric worse; I reverted it and shipped the uncompiled module. The README records three causes before any recompile:

  • False-positive penalty in verdict_metric was −0.25. The note says it needs to be −0.5. The current optimizer script still returns −0.25 for a true_product / not_a_product swap.
  • Training examples used URL references instead of the evidence snippets the model sees in production.
  • The train set lacked not_product rows that also have G2/Capterra listings — the Rhythm-style hard negatives.

The optimizer script has since been updated to derive examples from actual pipeline evidence. I have not published a new compile against that change. The freeze above is the uncompiled baseline.

Earlier the same day: search returned 400, eval kept scoring

Before this freeze, every Serper call returned HTTP 400 Not enough credits. Stage 1 was empty; crawl and the LLM still ran; five domains scored as empty misses. Those numbers are not published. The eval harness now probes GET /account and aborts if balance is not greater than 0, and it will not average a mid-run outage into the macro. The analyzer itself still fail-softs Serper 4xx into empty search. That is the remaining gap.

Reproduce

Labels, frozen predicted_products, and python3 rescore.py are in irfanalidv/stacksift-eval. The results file is eval_results.json. No API keys. The analyzer stays private.