RL Was Broken at Every Level - With Joseph Suarez (PufferAI)

In this episode, Joseph Suarez from PufferAI explains why he thinks RL never had an algorithm problem, but it had a code problem. Every part of the standard RL stack was running about a thousand times slower than it should have been, and once that got fixed, problems that used to take months started getting solved in seconds on one GPU. We talk about what makes a simulator good for RL, why most of their sims run on CPU, what he wants to do with scientific simulation, and why he open sources all of it instead of writing papers.
Key topics
• Types of RL and their applications
• Challenges in scaling reinforcement learning
• The role of simulators and hardware in RL
• RL in gaming: from chess to complex games like NetHack and RuneScape
• Future directions: scientific simulation and biological modeling

Chapters
00:00 - Introduction to RL and Puff AI
01:50 - Different settings for RL: Games, Robots, Finance
04:10 - RL in LM and other domains
07:00 - Challenges and solutions in RL scaling
09:55 - Building fast, efficient simulators
15:10 - RL for scientific research and simulation
19:57 - RL in complex games: NetHack, RuneScape, Dwarf Fortress
29:55 - Future of RL: Scientific discovery and beyond

Resources
Puff AI - Official Site - puffer.ai
NetHack - nethack.org
RuneScape - runescape.com
Dwarf Fortress - bay12games.com/dwarves
OpenAI Gym - github.com/openai/gym

Music
• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.