Sonnet 5 review: I ran 64 generations to find out if it's worth it

I’ve been testing every major frontier model release since the start of the year, and when Anthropic dropped Sonnet 5, I wanted more than a vibe check. I got tired of one-off tests I couldn’t repeat or compare over time, so I built something better: the How I AI Bench, a repeatable eval harness I constructed live using Claude Code while recording this episode. I ran Sonnet 5 blind against four other frontier models (Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro) across PRD quality, prototype generation, agentic task completion, and agent personality. The results were not what I expected.

What you’ll learn:
• What Anthropic claims Sonnet 5 improves over Sonnet 4.6, and where the benchmark data actually backs that up
• How I built the How I AI Bench in under 45 minutes using Claude Code, starting from my own stored session history
• Why I combined human vibe scoring (70%) with LLM as judge scoring (30%) instead of trusting either alone
• How to set up a local HTML scoring page so you can rate AI outputs on gut feel and export those scores as JSON
• Which model I recommend for PRDs, which for complex prototypes, and which for chatting with an agent daily
—
Brought to you by:
Runway (runwayml.com/howIAI)—The creative AI platform for images, video and more
Hyperagent (hyperagent.com/howiai)—Deploy fleets of agents that handle real work
—
In this episode, we cover:
(00:00) Sonnet 5 is out
(01:55) What Anthropic claims
(04:02) Why I’m done with one-off vibe checks
(05:05) Building the How I AI Bench live with Claude Code
(07:42) The scoring system
(10:43) Agent voice eval
(11:57) Quick recap
(13:58) Results: The How I AI index leaderboard
(21:21) What I’m improving for the next run
(22:16) Generating a Claire-weighted index
(23:53) Model-by-task recommendations
—
Tools referenced:
• Claude Sonnet 5: anthropic.com/news/claude-sonnet-5
• Claude Opus 4.8: anthropic.com/news/claude-opus-4-8
• GPT-5.5 (OpenAI): openai.com/index/introducing-gpt-5-5
• Gemini 3 Pro (Google DeepMind): deepmind.google/models/gemini/pro
• Cursor: cursor.com
—
Other references:
• SWE-bench Pro (agentic coding benchmark referenced): swebench.com
—
Where to find Claire Vo:
ChatPRD: chatprd.ai
Website: clairevo.com
LinkedIn: linkedin.com/in/clairevo
X: x.com/clairevo
—
Production and marketing by penname.co. For inquiries about sponsoring the podcast, email jordan@penname.co.