How to Avoid Two AI Catastrophes: Domination and Chaos (with Nora Ammann)

Nora Ammann is a technical specialist at the Advanced Research and Invention Agency in the UK. She joins the podcast to discuss how to steer a slow AI takeoff toward resilient and cooperative futures. We examine risks of rogue AI and runaway competition, and how scalable oversight, formal guarantees and secure code could support AI-enabled R&D and critical infrastructure. Nora also explains AI-supported bargaining and public goods for stability.
LINKS:
• Nora Ammann site (nora-ammann.replit.app)
• ARIA safeguarded AI program page (aria.org.uk/opportunity-spaces/mathematics-for…)
• AI Resilience official site (airesilience.net)
• Gradual Disempowerment website (gradual-disempowerment.ai)
CHAPTERS:
(00:00) Episode Preview
(01:00) Slow takeoff expectations
(08:13) Domination versus chaos
(17:18) Human-AI coalitions vision
(28:14) Scaling oversight and agents
(38:45) Formal specs and guarantees
(51:10) Resilience in AI era
(01:02:21) Defense-favored cyber systems
(01:10:37) AI-enabled bargaining and trade
PRODUCED BY:
aipodcast.ing
SOCIAL LINKS:
Website: podcast.futureoflife.org
Twitter (FLI): x.com/FLI_org
Twitter (Gus): x.com/gusdocker
LinkedIn: linkedin.com/company/future-of-life-institute
YouTube: youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw
Apple: geo.itunes.apple.com/us/podcast/id1170991978
Spotify: open.spotify.com/show/2Op1WO3gwVwCrYHg4eoGyP