AI Evals & Discovery

What you’ll learn in this episode:

• What “evals” actually mean in the AI/ML world

• Why evals are more than just quality assurance

• The difference between golden datasets, synthetic data, and real-world traces

• How to identify error modes and turn them into evals

• When to use code-based evals vs. LLM-as-judge evals

• How discovery practices inform every step of AI product evaluation

• Why evals require continuous maintenance (and what “criteria drift” means for your product)

• The relationship between evals, guardrails, and ongoing human oversight

Resources & Links:

• Follow Teresa Torres: ProductTalk.org

• Follow Petra Wille: Petra-Wille.com

Mentioned in the episode:

• How I Designed & Implemented Evals for Product Talk’s Interview Coach (producttalk.org/2025/09/interview-coach-evals) by Teresa Torres
Teresa’s - Interview Coach (learn.producttalk.org/course/story-based-custo…)

• Story-Based Customer Interviews (learn.producttalk.org/course/story-based-custo…) - On Demand course by Teresa

• AI Evals for Engineers and PMs (maven.com/parlance-labs/evals?promoCode=torres…) course (get 35% off through Teresa’s link) on Maven (maven.com)

• JSON (JavaScript Object Notation) (en.wikipedia.org/wiki/JSON)

• Anthropic (anthropic.com)

• The Product Leadership Wheel - A Framework for Defining and Growing Product Leadership at Scale (petra-wille.com/plwheel) by Petra Wille

• Lovable (lovable.dev)

• Behind the Scenes: Building the Product Talk Interview Coach (producttalk.org/2025/08/customer-interview-coach) by Teresa

• Previous episode: - Building AI Products (youtu.be/QYf3Y1Zb83Y)

Coming soon from Teresa:

• Weekly Monday posts sharing lessons learned while building AI products

• A new podcast interviewing cross-functional teams about real-world AI product development stories

Kapitelklicken springt direkt hin