08 - Analyst & PM playbook

Daily / weekly rhythms

Role	Cadence	Actions
Analyst	Weekly	Scenario funnel (impressions, views-as-clicks, carts, purchases); investigate CTR>100% or 0 impressions
Analyst	Ad hoc	ClickHouse recipes in 10; compare slug vs empty-slug volume
PM	When editing	Change one lever at a time; note slug + algorithm; QA 3–5 seeds in UI
Both	After train/reindex	Spot-check ML / vector scenarios for cold vs heavy users

---

“Expected item missing” checklist

1. Available? Catalog available / stock flags.
2. Filtered by nextQL? Temporarily loosen filter.
3. Diversity / purchase cap? Check category pile-up and user’s purchase history.
4. Score threshold (vector)? Too high → empty neighbors → category/min fill.
5. Wrong logic/algorithm? Scenario slug on the placement.
6. Seed quality? Bad/missing embedding for seed.
7. Timing? Just indexed? Wait for Qdrant/CH catch-up.

---

Interpreting metrics safely

Observation	Likely cause	Action
CTR ≫ 100%	Views with slug ≫ impressions	Audit host trackView / attribution (09)
Impressions ≫ 0, clicks ≈ 0	No slug on views; or users don’t open PDPs	Check SDK click/view tagging
Carts/purchases tiny	Funnel real or events not sent	Verify trackCart / trackPurchase
Rail looks generic	Cold start / fallback / missing factors	Check algorithm + training; identity merge
Sudden empty rails	Filter too strict / threshold / outage	Admin preview + engine logs / Qdrant health

Remember: Admin “clicks” = type=view + slug.

---

Scenario edit recipes

Goal	Lever
Hide OOS / wrong region	nextQL filter
Prefer featured / margin SKUs	booster (soft)
Stop one category dominating	maxPerCategory
Show newer catalog	freshnessBoostDays
Discoverability for obscure SKUs	injectLongTail + percentage
Fewer repurchase suggestions	purchasedItemsMaxPercent + pass userId
Stricter similar	raise score_threshold
Richer cart rail	multi-seed-hybrid + adaptive

Change one of {filter, boosters, constraints, algorithm, qdrant} per experiment when possible.

---

Experiments (high level)

If the tenant uses Admin experiments / variants:

• Keep scenario_slug consistent on impressions and attributed interactions.
• Don’t change instrumentation mid-test without splitting analysis windows.
• Primary metrics: impression volume, attributed view rate (once instrumentation is correct), cart/purchase rates.

---

Handoff questions for engineering

• Is placement using scenario slug X end-to-end?
• Are impressions fired on visibility or on render?
• When is trackView called with a scenario slug?
• Is userId passed into recommend calls for purchase cap?
• Last successful train / reindex time for ML or vectors?