Priya Nair
“The spec used to be the contract. Now the eval suite is the contract.”
- Martin Knapic, Product MoatPriya Nair joined product from a data science background, which shows in how precisely she talks about what 'ready to ship' means for a probabilistic feature. Nine years in, most of them building AI-facing products, she's become someone I go to for a straight answer on what an eval suite actually replaces in the PM job - not marketing language, the real mechanics.
Identity & Background
- Name
- Priya Nair
- Current role
- Senior Product Manager, AI Platform
- Current company
- Northlight
- Location
- Bengaluru, India
- Years in product
- 9 years
Your story: how did you end up in product?
I was the data scientist stuck explaining model results to a PM who'd then explain them badly to the exec team, twice removed from the actual finding. I got tired of the game of telephone and asked to own the roadmap conversation directly. Turns out I liked owning the trade-off more than I liked owning the model.
Core PM Philosophy
How do you define product success?
Whether customers' downstream teams stop double-checking our output. Adoption is a proxy; reduced verification effort is the real signal that a retrieval product is actually trusted.
What's your decision-making philosophy?
Three gates: the eval score is stable across two consecutive model versions, the failure modes are boring rather than surprising, and support can explain a wrong answer to a customer in one sentence. If any of those fail, it's not ready, no matter how good the demo looked.
What's your biggest PM lesson or mistake?
Reflexively de-risking. In deterministic software, shipping the safe version first was almost always right. With generative features, the safe version is often the boring one, and you learn less from it than from a bolder version with a tight fallback.
How do you manage stakeholders?
That 'make it more accurate' isn't a real request without a metric attached, and that I need their help defining that metric before we can even scope the work. The best AI-era engineering partners I've had treat evals as a joint deliverable, not a QA afterthought.
Product & Decision-Making
How do you do customer discovery and validation?
We run structured blind comparisons - customers rank answers from our system against their current one without knowing which is which. It's uncomfortable and far more honest than a satisfaction survey.
AI & the Future
How is AI changing product management?
I used to write specs that described exact behavior. Now I write specs that describe a distribution of acceptable behavior, and the eval suite is what actually enforces the contract with engineering. If I can't define 'good enough' in a way a script can check, I haven't finished the spec.
Which AI product do you use and respect?
Claude and a good notebook environment. I also keep a couple of open-source retrieval libraries around purely to benchmark against; respecting what competitors ship for free keeps me honest.
What's one thing about AI everyone gets wrong?
That hallucination is the main failure. In retrieval, the quieter and more common failure is confidently returning the second-best document. It looks fine in a demo and quietly erodes trust.
Beliefs & Opinions
What's your unpopular product opinion?
Eval literacy outside the AI team. Legal, support, even sales are all now affected by model behavior they can't independently assess, and most orgs haven't built the muscle to read a confusion matrix outside of engineering. That gap is going to cause real incidents before it gets fixed.
What makes a great product?
Something whose failure modes are boring. A great product doesn't never fail; it fails in ways users can predict and recover from.
Company & Product
What does your product solve?
We're the information-retrieval layer other software companies plug into instead of building their own search and RAG stack - the part of the product that answers 'find me the right document' correctly, at scale, across messy enterprise data.
What's the market opportunity?
We don't compete on 'can you build this' - anyone can build a version. We compete on 'can you keep it accurate as your data grows and stays messy,' which is the unglamorous, compounding work most internal teams underestimate and eventually get tired of maintaining.
More from the conversation
What's a hobby that's more connected to your product instincts than people would guess?
Competitive badminton, which I still play at a decent club level. It's a game of small, fast corrections under uncertainty - you don't get to plan three shots ahead, you read the racket angle and adjust. That's most of my actual job now too.
Was there a moment you almost left product?
Early on, yes - a year into a role where I was purely a backlog administrator, no real influence on strategy. I moved teams instead of moving industries, and it was the right call. The title wasn't the problem; the scope was.
Who's the actual customer, and how big is your world inside Northlight?
Our ICP is a mid-to-large B2B software company that has search or RAG on its own roadmap and would rather license a good one than build a mediocre one in nine months. I run a team of six PMs and about 30 engineers across the platform, inside a company of roughly 400.
Where do you see your corner of the industry heading?
Retrieval quality becomes the actual moat, not the model. Everyone has access to roughly the same foundation models now; the products that win will be the ones with the cleanest, best-curated, best-retrieved data underneath them - unglamorous infrastructure work, decisive advantage.
One thing you'd tell a PM starting out today?
Learn to read a confusion matrix before you learn to run a stakeholder workshop. The workshop skills transfer from any era of software. The measurement skills are what's actually new.
Enjoyed this conversation?
A new interview with a product person lands every week. Get it in your inbox.