RAG Won’t Save Your Messy Data: The Brutal Truth About AI Reliability

SUMMARY: The RAG (Retrieval Augmented Generation) pattern is one of the most frequently used to augment LLMs with context-specific information. Let’s explore RAG.
GUEST: Roie Schwaber-Cohen (linkedin.com/in/roiecohen), Head of Developer Relations at Pinecone (pinecone.io)
SHOW: 1018
SHOW TRANSCRIPT: The Reasoning Show #1018 Transcript (docs.google.com/document/d/1Hu619Y2UO5Syootvlw…)
SHOW VIDEO: youtu.be/-kZZEMR341Q
SHOW SPONSORS:
• Nasuni - Activate your data for AI and request a demo (nasuni.com/AI)
• ShareGate - ShareGate Protect. Microsoft 365 Governance, we got this! (sharegate.com/protect)
SHOW NOTES:
Topic 1 - Welcome to the show. Tell us a little bit about your background, and what you focus on these days at Pinecone
Topic 2 - Let’s begin by talking about RAG systems. What are they? Why do companies choose to use them? What benefits do they provide in AI systems?
Topic 3 - At a high level, RAG sounds straightforward—retrieve relevant context, generate an answer. But in practice, where does it break first as systems scale?
Topic 4 - I’ve heard that RAG systems can return answers that are technically correct but fundamentally wrong. What’s a concrete example of that happening in production—and why does it slip past most teams?
Topic 5 - In traditional systems, we assume there’s a single source of truth. But in enterprise environments, ‘truth’ is often versioned, contextual, and conflicting. How should teams rethink ‘truth’ when building AI systems?
Topic 6 - A lot of teams assume their knowledge base is ‘good enough’ for RAG. What do they usually underestimate about the messiness of real enterprise data?
Topic 7 - There’s a growing narrative that better reasoning models can compensate for weaker retrieval. From what you’ve seen, where does that idea fall apart?
Topic 8 - If correctness depends on things like timing, policy scope, or configuration, how should teams design systems that understand context—not just content?
Topic 9 - Looking ahead, what replaces today’s RAG architectures? What patterns are emerging among teams that are actually getting this right?”

FEEDBACK?
• Email: show @ the enterprise ai show dot com
• Bluesky: @TheEntAIShow.bsky.social (bsky.app/profile/TheEntAIShow.bsky.social)
• Twitter/X: @TheEntAIShow (x.com/TheEntAIShow)
• Instagram: @TheEntAIShow (instagram.com/TheEntAIShow)