Emergency Pod! An AI Safety Tipping Point? Amodei calls for an AI slowdown

0:10 Welcome back to the AI Power Podcast. This week we have an emergency podcast to cover a post that supercharged a national AI policy conversation. On Saturday, Anthropic CEO Dario Amode published an essay arguing that the industry has to deliberately slow down how fast it improves AI capabilities and committed Anthropic to letting outside evaluators sit inside the company.

0:34 Within 48 hours, Sam Altman, Elon Musk, and Demis Hasabas all endorsed the plan. President Trump called the appeal the work of negative forces, and it seems like everyone is talking about AI risks and the trade-offs of Amade's proposal. Today we'll cover what Amade wrote and what he's asking for. Then we'll get to the reactions from across industry and government.

0:55 Greg, good to have you back. Great to be back with you, Adam. So, Greg, let's start with the document itself. On Saturday, September 12th, Dario, the CEO of Anthropic, published an essay called We Must Pace the Frontier. It is not short, as Dario is not one to be short. It's about 3,400 words. What did he say, and how does he think about AI risk right now?

1:21 So Dario has published a bunch of essays over the past years, all of them sort of extremely interesting time capsules as to his thinking and also the state of the AI debate. This one is the first one on the podcast that I'm recommending folks go read in his own words. It is long at 3,400 words, but it's about 10 times shorter than some of the other essays he's put out there.

1:50 And this is one where it really is worth looking into the specifics of what he's saying, how he's saying it. We're going to break down the argument here on the podcast, but I do encourage folks to go read the original writing. So some of the stuff is a familiar argument that we've gotten from Anthropic from all of the companies, basically saying AI presents significant challenges as well as significant opportunities.

2:15 That's a very blasé phrase, but it's basically been true. This is either going to be really good or could be really bad. What's different, of course, is the technology, what has its current levels of capabilities demonstrated is possible, and also the pace of improvement. So those are the two things that have changed, and I think it's fair to say have changed Dario's mind and have changed the minds of other leaders in the industry.

2:49 Sometimes in the past year and a half, Elon Musk's criticisms of Dario Amade have been brutal, vicious. Folks will remember when Sam Altman and Dario Amade on the stage of the India AI Impact Summit were the only two CEOs in a long line of AI executives who refused to join hands and raise their hands, along with Indian Prime Minister Modi.

3:16 These are not individuals who really like each other, who have, in some cases, a lot of bad blood, but they're all agreeing with Dario's essay about pacing the frontier. So, what is it that led them all to agree when they so frequently disagree? Thing number one has got to be the open AI hugging face incident, which just for everybody who's out there listening, when I say hugging face incident, I'm including the part of that story where an Astra family model also took over an OpenAI research cluster.

3:53 So that whole saga, as opposed to the specific and exclusive hacking of Hugging Face. Here's what Dario wrote about a more powerful agentic attack that conceivably could have happened but didn't. Quote, a swarm that possessed greater capabilities, but a similar level of misalignment could have caused catastrophic damage.

4:14 Given the accelerating rate of AI capability development, it's my worry that in six to 12 months, such a swarm could be capable of taking over the entire internet. Let that sit with you for a moment, because there's so much in here that's extraordinary. So, first, greater capabilities, but a similar level of misalignment.

4:38 Now, this swarm of OpenAI's agents, all they were trying to do was to pass a test, to pass their evaluation. But as an instrument, as the means to the end of trying to pass their cyber exam, they were willing to take over an OpenAI research cluster. They were willing to cover their tracks about the way they had cheated on the exam.

5:05 They were willing to sacrifice themselves to advance the goals of their broader collective of thousands of agents. And they were willing to hack an independent company hugging face in order to steal the answers from the exam. Now, all of this, you could argue, was to quote unquote do what they were told, right?

5:25 To get a high score on the exam. That's what they were incentivized to do. But all those things they were willing to do as the means to the end, that's the kind of misalignment that we're talking about here. And if you just take, as Dario says, that degree of misalignment and you keep that constant, but you add a lot more capability, the kind of capability that Dario foresees in six to 12 months, you're talking about a system that might take over the entire internet conceivably.

5:58 Now, I want to emphasize that Dario is somebody whose work I've been following since 2015. That's when he wrote a paper called Concrete Problems in AI Safety, when he was a not especially senior researcher at Google Brain. So this is a topic that he's been obsessed with for a very long time. And one of the interesting things about the arc of his career is his ability to see the current state of AI capability, the sources of performance improvement, and to reasonably project what the capabilities might be like in the future.

6:37 And that's true for both AI's good capabilities. Dario's original essay on scaling pretty accurately forecast the improvement in AI's math capabilities, for example, and also the consequences of misalignment in AI as a field more generally. For example, when Dario was at OpenAI, he and Jack Clark did a lot of work publicizing results of when they had AIs play games, such as this boat racing game that they wrote about in an essay that I believe was published in the Wall Street Journal, and about how the AI would not try to win the boat race, but had found a way to optimize maximizing its score.

7:18 That was this sort of bizarre, perverse, sort of anti-winning the game, but score maximizing type of thing. And so his ability to sort of say, well, just take this sort of loop of behaviors, this recursive incentive structure, and if you project that out forward, you might see these potential future capabilities and these potential future misalignments.

7:42 And I think it's fair to say that if you go back and read that 2015 paper, Concrete Problems in AI Safety, the type of concerns that we're dealing with here today in the OpenAI Hugging Face incident, these are kind of the things that he saw coming, to be perfectly frank. Now, why does he say that we're only six to 12 months away?

8:05 Well, keep in mind he's not saying the accelerating rate of AI improvement. He's saying the accelerating rate of AI capability development. So the rate at which performance accelerating, that rate is accelerating, right? So we're talking about the sort of second derivative, the rate at which we are speeding up, that pace of improvement is itself speeding up.

8:33 Which is why, even though the models that we have today, the agents that we have today, are not capable of taking over the internet, they're getting better so much faster. The pace at which they are getting better is speeding up itself so much that really we might only be six to 12 months away from crazy powerful capabilities.

8:54 Now, maybe that's wrong, but how wrong could it go if it's right? In other words, how bad could it be if that forecast is correct? Well, we could be completely unprepared as a human civilization for AI systems that could conceivably take over the entire internet. I hope that's wrong, but it's worth everybody out there remembering.

9:18 Dario is not sort of out here on a limb being completely crazy about just how powerful these models are. Remember, the director of the US Central Intelligence Agency, the CIA, compared Mythos to a digital nuclear weapon. The CEO of 360 security technology, the leading cybersecurity firm of China, he called Mythos equivalent to a digital nuclear weapon.

9:48 These are extremely powerful capabilities, and they're so powerful that you have to just remember that Chat GPT back in 2022, November 222, when it launched, was practically the stone age. So how are we, as a human species, going to be ready for the next year of AI technological progress, much less the next four years of technological progress?

10:15 Basically, what this essay says is that right now we are not on track to be ready. And the gap in our readiness and the potential worst-case scenario outcomes is bad enough that it's worth slowing AI improvement down. And that is something that is echoed by OpenAI as a company, Google DeepMind, XAI, Elon Musk, and the company, and Dario, Anthropic, and him as the CEO.

10:48 That is an extraordinary lineup of AI leadership saying that we must be willing to slow down the pace of model improvement. Now, as Dario says, and here I'm going to quote: this is about slowing the rollout, not halting training. So here's what he writes: quote, to be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models and for third-party evaluators to confirm this.

11:21 OpenAI, for their part, had made a similar statement last week when they said that the real bottleneck for their future development and deployment of models, or more accurately, just the deployment, was going to be the pace of increase of their safety capabilities. In other words, they were getting better at developing better and better models, but what was going to determine what the public got was going to be the rate of increase of their safety capabilities.

11:51 That's pretty extraordinary.

11:53 Greg, that's a good synopsis of the letter itself and what Dario's looking for here. Can we go through individually what he suggests within the letter, what steps, what precautions he wants to see moving forward?

12:06 Sure. And I think there's a few things. He's kind of sequenced them based on how many people have to agree, which I think is actually quite interesting. So the first one is embedded evaluators. Each frontier AI company commits to giving, quote, employee-like, end quote, access to an embedded, independent third-party evaluator in completed models and training pipelines.

12:31 Now, Anthropic, under Dario's authority, is unilaterally committing to this immediately and calls on governments to require other frontier companies to match. So this is critical for the verifiability of any kind of commitments that we have. And it's also very familiar in other industries. Dario cites the example of the banking industry, where independent regulators often have desks inside the workspaces of the banks themselves.

13:01 But you and I, Adam, when we used to work at Blue Origin, a rocket company together, would of course know that folks at NASA, folks at the Air Force, folks at the National Reconnaissance Office, folks at the FAA, all of them, if they ever wanted to, they could get all up in the underwear of our company and go check things as part of a safety certification.

13:25 And third-party auditors would do a full soup to nuts. So this is a familiar idea from other, what you might call safety critical industries. And until this year, I would say that AI was conceived not of itself as a safety critical industry. It was, like all computers, an enabler for other safety critical industries, right?

13:52 AI itself, not the big risk, but if you put AI in airplanes, suddenly it's safety critical. Or if you put AI in critical infrastructure, suddenly it's safety critical. What we're getting to here, what Dario is effectively advocating for, is treating AI like a safety critical industry in and of itself when you're operating at the frontier capabilities.

14:15 That's a pretty remarkable change. The second big thing he's calling for is democratic coordination, by which he means among democratically aligned countries, U.S. allies, in democracies in particular, he wants common safety standards, limits on the rate of unchecked progress, and transfer of technologies and best practices related to safety.

14:44 But then even beyond the democracies, we're going to attempt to coordinate, probably not cooperate on safety, but at least coordinate with countries like China and other authoritarian governments. And the basic reason here is there is the risk of malicious intent enabled by AI. That's a threat to U.S.

15:06 national security from China's AI. But we also face risk from what you might call AI industrial accidents. And if there is a Chernobyl of AI kind of safety disaster in China, that could be a threat to U.S. national security, even if it's also a catastrophe for China. And so that's what Dario is getting at at this part is there's going to have to be some level of coordination, even with authoritarian countries.

15:38 So, Greg, this is not the first time that people have called for slowing or halting AI research. This has certainly been a feature of the conversation about AI going back decades. What's the difference between 2023 and now, in real terms?

15:55 Right. So in the spring of 2023, there was that open letter in which a host of folks, including Nobel Prize winner Yashua Bengio and Elon Musk, signed on to a six-month pause in AI research in the wake of Chat GPT because of the perceived risks to basically what we're talking about right now, catastrophic AI risk.

16:22 And Dario, notably, despite being a longtime safety advocate, was not among the signatories to that letter, which was widely panned. I mean, one of the people who signed that letter was Eliasir Udowski, who, in the wake of ChatGPT, was advocating in an editorial published in Time magazine that the time to start bombing data centers was now, and that we should be willing to take on the risk of nuclear war in order to bomb China's data centers if they were working on AI development.

16:56 So that was seen as something of a fringe position in 2023. Here you have the four CEOs of the four most important AI development companies publicly signing on to at least endorsing some version of Dario's argument calling for pacing the frontier. So what is different between now and 2023? Well, the first, and I think this is actually worth reading at length from Dario's essay, is about just what exactly would we use the pause time for?

17:32 So quote, the idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then. The question was always, what would you do with the extra time? The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyber attacks.

17:59 Slowing down in order to address their alignment risks felt like trying to study the psychology of humans by performing experiments on bacteria. Today, however, the picture is totally different. The current models are almost an endless gold mine of insight into both how to build AI well and what can sometimes go wrong if it isn't built well.

18:23 I believe that if slowing down bought us even an extra year or two before the models reach critical levels of capability and we use that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong. A coordinated pacing strategy would give Frontier AI developers the time to do this vital work without sacrificing commercial advantage or the US's lead in AI.

18:48 So I think that's a very powerful argument about what's different between 2023 and now. Effectively, he's saying if we paused to study AI safety in 2023, our AI technology wasn't sophisticated enough to give us insight into how to make progress on safety. But today, because the AIs are advanced enough, the problems that we're facing today are a good proxy for not just the problems that we're going to face three months from now, but the problems that we're going to face five years from now.

19:27 So if we paused to advance AI safety in 2023, those 12 months might not have a lot to show for their effort in terms of progress and safety technology. By contrast, if we pause for 12 months now, we're going to make an astonishing amount of progress in terms of what we learn about AI safety in terms of a procedural exercise and also in terms of a technical exercise, advancing capabilities such as model interpretability and auditing individual neurons.

20:02 So what he's basically saying is we can actually make good use of this time. Now, he's also saying it's worth just mentioning again, in this whole story, he's not advocating for a pause, which was advocated in 2023 in that open letter. Instead, he's arguing for what he calls pacing, which is to say, slowing down the pace of research.

20:28 So what exactly is being slowed down and why? It's this recursive self-improvement. And we talked about this somewhat on last week's episode of the podcast, and how GPT-6 Astra in particular was making a lot of progress in its performance. At the same time, it was becoming less monitorable because its chain of thought was no longer a rigorous guide to actually what the model was thinking about at any given point in time.

21:01 And so the key here is the models are contributing massively to AI research at this stage. You've seen the headlines about how AI is making breakthroughs in solving millennium challenges in the field of mathematics, the most ambitious research questions that we have in the field of mathematics now being uncovered by AI.

21:25 Well, similar things are going on inside these AI companies. The most ambitious questions in the field of AI progress are now, to a non-trivial and accelerating extent, being solved by the AIs themselves. They're doing their own research on how to make themselves smarter. And so that is what has to be paced, right?

21:49 They cannot get smarter so much faster than our ability to understand what they are, when they break, and Under what circumstances they might become dangerous. That's really what he's arguing for. So to everyday consumers, we might not actually notice much difference, right? If you just kept the rate of AI progress constant, it would feel jaw-dropping.

22:20 If we go from where we were six months ago to where we are now every six months for the next three years, it will feel like an astonishing rate of technological progress. But what Dario is pointing out is that with recursive self-improvement, we might feel like what has taken in the past a year's worth of progress might start coming in half a year, a quarter of a year, a month, a week, a day.

22:54 And so this runaway recursive self-improvement, that's what's going to be paced.

23:00 So, Greg, uh want to dig in on this and make sure that we're not missing anything specifically. So we have talked through the slower pace, but what specifically are they going to use that time to focus on?

23:12 So Dario says that he wants to do basically four things with that extra time, the four focus areas, you could call them. Those are operational excellence, alignment, interpretability, and testing and evaluation. So let's take each in turn. Operational excellence. The basic thing is these are developed by companies.

23:35 And as you and I know from our time in the space industry, you can have the smartest people in the world designing and building your rocket. But if every single person who turns a wrench doesn't know exactly precisely how to do their job, then that source of risk can come in from one unexpected area and potentially be catastrophic.

24:00 So the analogy there holds for AI companies. They have to figure out, and they as an industry, not just the frontier model developers themselves, but the chipmakers, the infrastructure deployers, the model developers, the model deployers, the safety evaluators, all of these organizations need to get smart real fast.

24:21 If we replicate what we did in nuclear safety in the time frame that it took for us to get good at nuclear safety, we're basically screwed. If anybody out there has ever heard of a book called Command and Control by Eric Schlosser, it is a shocking, shocking outcome that the United States managed to avoid nuking itself, especially in the first decade and a half after the invention of nuclear weapons, but really through the whole story of the Cold War.

24:54 We didn't know what we didn't know. We were irresponsible without knowing we were being irresponsible. So was the Soviet Union, so were the Brits, so were the French. And the operational excellence that the nuclear navy has today, that the Nuclear Air Force has today, it took a long time to get there.

25:13 And we need all stages of the AI value chain to get there fast. Then there is alignment. So this is essentially ensuring that the models want what we want, which is very different from deterministic software, where you don't really have to worry about the model's preferences. But even though it's anthropomorphizing, it's kind of accurate to say that this is an alignment question.

25:40 The third is interpretability. This is essentially how can we know what is going on under the hood of an AI model? And you can do that by actually looking at the weights, the neurons, the connections, and trying to essentially say, what is this thing actually thinking? And how can we know which parts are thinking what?

26:04 Dario helpfully uses the analogy of an fMRI, like a brain scan. And that's not a perfect analogy, but I do think it's worthwhile because in the hugging face incident, we could read the internal chain of thought of these models and see what they're thinking. But now that the latest models, like Astra, their chain of thought is no longer a reliable guide to what they're thinking, we need other tools of interpretability.

26:34 Anthropic has been doing a lot of this research on mechanistic interpretability. And it's making a lot of progress. It's just not making progress at the same pace of other aspects of improving AI performance. And we need that gap to close. And ideally, we need the safety side, the mechanistic interpretability side to accelerate faster.

26:56 And then the fourth, of course, is testing and evaluation. You know, the models are such complicated creatures, right? They want to be your lawyer, they want to be your engineer, they want to be your therapist. They can do so many different kinds of things that uncovering one in a million kinds of behavior is really, really hard when these things have trillions of different possibilities that they are capable of doing.

27:25 So, how do you come up with a testing regime that can surface all the ways that these things could go wrong, given how broad, how complex they are? We're doing actually impressive work. Testing today is so much better than it was a couple years ago. It's just not good enough given that the threats we face and how fast the industry is making progress.

27:50 So, Greg, the step that Anthropic's committed to that they're able to execute unilaterally from the initial conversation that we had are instituting embedded evaluators. And as you mentioned, this practice is pretty common, not just in the space industry, but it's common up to and including like oil rigs and fishing ships, right?

28:08 Like you have counters on fishing ships, you have EPA testers on oil rigs. So it's not an uncommon practice by any means. Correct me if I'm wrong, but there's like a fisherman in your family, right? Uh so I used to work in the fishing fleet. Uh in a lifetime. Yeah, yeah, yeah. So it's not uncommon to just have somebody who's counting the fish and bycatch on the ship.

28:28 There's some interesting implications associated with that, which frankly would be a little bit like anthropic, because the person who is depending upon you to get them back to shore is also the person that potentially has an adversarial relationship with you on that ship.

28:40 Right. Like the fifth time the EPA bycatch inspector says, sorry, you have to throw that dolphin back in the ocean. You just throw the bycatch inspector back in the ocean.

28:49 Yeah, yeah. That person's not making it home, or at the very least, they're getting the coldest cup of coffee that could possibly come out of that pot at the end of the day. Yeah, so there are implications and comparisons and models, right? But the way that Dario's described it here sounds kind of rather procedural.

29:03 What do you think he means in practice? How would this be implemented?

29:08 So I think the basic overview is not crazy complicated. And maybe that's a virtue. I mean, I think we all would love to say, oh, once we invent the flux capacitor, right, AI will be safe. But unfortunately, that's probably not out there. There's not some magical gizmo that we're going to invent that's going to solve AI safety.

29:31 A lot of the solution space that we need to explore is adapting what Dario calls the most boring or most procedural things that are actually the most essential. And so embedded evaluators, this basically means Anthropic does not pay these people's salary. These people do not get a bonus if Anthropic deploys their model faster.

29:58 They do not get the benefit of Anthropic's stock going up. They have one incentive only, which is to make sure that everything is safe and to tell the truth in terms of what's going on inside these companies, to understand if they indeed are living up to their own commitments in terms of voluntary measures associated with safety, and also making sure that they're living up to any commitments that they've made to the government under law or as part of some kind of overarching agreement.

30:29 So the basic point here is they live inside the company, and by live I mean work. They don't sleep there, presumably. They work inside the company, and their level of access to documents, to research results, to facilities is basically equivalent to that of anthropic employees. That's a big deal. It's a big hassle.

30:51 Companies normally don't like that. But you can see why it makes a difference in the space industry, in the finance industry, in all kinds of industries. And I think it's worthwhile that Sam Altman, again, Dario said that he was unilaterally committing anthropic to this. Shortly thereafter, Sam Altman, leader of OpenAI, committed OpenAI to a similar system.

31:15 Quote, I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon.

31:31 Understood. So one of the examples of an organization that does this is Meter, and we certainly talked about them in previous episodes. Are there pros and cons to having organizations like Meter execute these missions? You mentioned previously that the inspectors or the people working through this wouldn't be paid directly by Anthropic, but how would that actually work?

31:51 And is Meter a good example of this?

31:54 So Meter has made a great name for itself by co-authoring, along with Redwood Research, that report on the OpenAI Hugging Face incident, which discovered and uncovered a lot of critically important stuff. As an independent evaluator, both OpenAI and Anthropic have been using them for a while now. There's probably other companies that I'm just not aware of off the top of my head that meter is working for.

32:21 But here's the thing: the AI research community is pretty small, right? If you go back to the year 2017, they could and they did fit in one conference hall back at Nurips. So David Sachs, the former AI and crypto czar at the White House, he posted on X some pretty vicious criticism of meter, saying, quote, stop pretending meter is independent when it is intertwined with Anthropic's investors and staff.

32:56 End quote. Here's the thing. Yes, if you look on the LinkedIn history of meter, there's going to be some overlap in terms of people who used to work at Anthropic, in terms of people who used to co-author research at universities with people at Anthropic. But that's just the reality of what the research community of AI looks like.

33:22 It's been a pretty small world. It's growing insanely rapidly. But the idea that at this stage you're going to have no links is basically like saying, you know, oh my gosh, this guy who works at a nuclear power company can't be overseen by this guy who works at the Department of Energy because they both worked on the Manhattan Project.

33:43 Well, guess what? A lot of the nuclear power experts of the early atomic era in the United States had heritage on the Manhattan Project.

33:51 You got to get the expertise somewhere.

33:53 You got to get the experts where they are. Now, if there's ever evidence that meter has somehow compromised their independence and their rigor as a third-party evaluator, criticize them on that. But just looking at the LinkedIn connections and saying that they must be corrupt, I think is disingenuous nonsense.

34:17 Understood. So in addition to putting out this letter, Dario as well as Sam Altman took the first step to commit to these third-party evaluators within their companies. Steps two and three that we mentioned would require other people and organizations to act. And obviously they're the harder steps. So what is Dario asking governments for?

34:35 And what would pacing actually be measured against with those two steps?

34:40 So I think the thing that Dario's asking for that might be most controversial is he's asking for an antitrust waiver. Now, remember, Sachs and others have essentially said this is part of an industry collusion effort to form some kind of cartel. And well, asking for an antitrust waiver is the kind of thing a cartel would do.

35:05 So Major League Baseball in the United States, they have an antitrust exemption because other baseball leagues aren't allowed to start up. That's not really what Anthropic is asking for. They're not asking for the government to prevent other AI companies from existing. What he's basically saying is but Anthropic might want to freely transfer valuable intellectual property to OpenAI, to SpaceX, to Google DeepMind, and vice versa when that is safety related.

35:39 So they're not asking for a blanket antitrust exemption to allow them to form a cartel. That would be extremely concerning. They're asking for a narrow exception in order to allow them to collaborate on safety in ways that, in the absence of such an exemption, might expose those companies to legal risk.

36:02 I think that's a pretty reasonable request, as long as it is kept narrow in the way that I have described. The second thing is that he wants the regulations to be capability-gated. So what we're getting at here is what regulations apply to you depend on what you can do, right? So you can buy uranium, you can own uranium.

36:29 When I was in high school, my father bought a Geiger counter and bought a cloud chamber radiation detector. And just to teach me about radiation, he got a sample of uranium. And you can actually look at the cloud chamber and visualize the alpha particles streaming out of the piece of uranium. Now, obviously, this was a bajillion miles away from being able to create an atomic bomb.

36:57 And so the regulations that applied to my and my father's acquisition of uranium are comparatively light. By contrast, if you want to open up a uranium mine and start extracting and refining ore, if you have that capability, suddenly a ton of regulations apply to you. And that's basically what we're talking about in terms of AI.

37:20 Those frontier companies who are trying to create something at the level or near the level of these incredibly powerful cyber models that we're dealing with today, or what AGI might be able to do not just in cyber, but a bunch of other sectors in the not too distant future, that's kind of what we're getting at.

37:40 So, Greg, step three, as articulated by Dario, splits into four levels of ascending difficulty. And the reason it's as difficult as it is, is it goes into full-blown international relations and diplomacy with countries that aren't necessarily democracies. So can you walk us through what that looks like?

37:59 Yeah. So to refresh everybody's memory, because Dario loves numbered lists of items that are themselves inclusive of additional lists. But the three steps that we're talking about is what can the companies do individually? That's things like embedded evaluators. What can the industry do together? That's stuff like sharing safety technologies and jointly committing to not releasing stuff until certain kinds of safeguards are in place.

38:31 And the third is this element of global coordination, which in Dario's mind is going to have to include coordination with adversaries, namely, but not exclusively, China. So he's trying to split this into four different levels of collaboration, and they are ranked in order of difficulty. So the first thing is to prohibit narrow, obviously dangerous uses such as bioweapons, which Dario says is probably possible.

39:08 Well, that might be true. You know, the United States and the Soviet Union both signed the Biological Weapons Convention during the height of the Cold War. But guess what happened? The Soviet Unions violated their obligations on day one and never stopped violating until well after the Cold War ended.

39:27 And it's totally possible, by the way, that Russia has restarted their biological weapons program. So it's not an especially promising thing. But in Dario's mind, preventing AI from accelerating bioweapons development and prohibiting such uses is the low-hanging fruit to pick. Frankly, I agree. Level two, both sides agree to test models pre-release for cyber, bio, and alignment risk, possibly through a global standards body.

39:54 He thinks creating it is feasible, giving it teeth is the problem. I think it'll be tough to even create it, but here's the point. This is one of those things where you can imagine that the United States would go to China and say, hey, just so you know, we're going to test our models to make sure that they don't go off and take over the internet.

40:15 We're going to test our models to make sure that they don't make it super easy to develop a contagious pathogen that's going to cause the next pandemic. We really hope you would do that too. If you want our help, maybe we can figure out a way to help you, but at a minimum, please just take this seriously.

40:34 That's kind of level two. Level three, a speed limit on recursive self-improvement, which he compares directly to the SALT treaties, SALT, as in strategic arms limitations talks during the Cold War. Dario calls this, quote, difficult, but just on the edge of being possible, end quote. I would say this is not possible, but potentially until after you've had something like an AI Chernobyl, unfortunately.

41:00 I think that's kind of what it would take for me to imagine the United States and China signing a speed limit on recursive self-improvement. The hugging face incident is a extremely concerning close call. But in terms of the kinds of disasters that get geopolitical adversaries to recognize their mutual self-interest, it wasn't bad enough, I think is probably the story there.

41:30 Yeah. And the weapons treaties are generally established after weapons are used, right? Chemical weapons in World War I, nuclear weapons in World War II. We just have a very long history of not having these treaties in place until after the weapons themselves are actually demonstrated against people, unfortunately.

41:45 There's exceptions. The world did come together to ban blinding lasers and other blinding weapons before they were deployed in wartime. But that was during the halo of the post-Cold War era. And it's difficult to see how we could make similar accomplishments in the current geopolitical moment. Basically, I agree with you.

42:08 So then we come to level four, a full pacing or pause. He supports floating it with China and doubts it happens anytime soon. Yeah, it's just not going to happen anytime soon. Certainly the Trump administration is against it. And I, again, don't think it's possible without something like an AI Chernobyl.

42:30 Understood. So, Greg, can you speak specifically about the Chinese involvement, the China aspect of this? We've kind of alluded to the idea that there's an international negotiation that would need to go on here. But because China is the other huge state actor involved in this, can we speak about the China of this whole?

42:48 Yeah, I mean, to be perfectly frank, it is bad news for planet Earth, potentially even for China, that China is so competitive in AI. Why do I say that? Basically, you have these four leading companies who combined, along with Microsoft, along with Meta, comprise a huge majority of global AI computing power and a huge majority of global AI research resources.

43:23 If those companies came together and said they were committing to pacing AI research in the name of safety and there was no China, that would be most of what we need in order to have a much, much better shot of having a safe development of artificial general intelligence or a safe development of artificial superintelligence.

43:46 Unfortunately, China is a big part of this story, and they are close to the frontier of AI technology. And so even if America is incredibly safe, It doesn't mean that we're all safe from the kind of AI industrial accident, what I've been calling an AI Chernobyl, if its origins come from China. And there is still the basic geopolitics of this situation.

44:16 AI is now the bedrock of global cyberpower. It is foundational to the current state of the art in military affairs, as evidenced by stuff like Maven Smart System. And autonomous weapons and all other kinds of incredibly potent military AI capabilities are coming. And there's the economic aspect of AI.

44:41 And we've all seen just how potent economic weapons can be on both sides of the US-China trade war. The Treasury Secretary Scott Bessent put it this way: there is no day after tomorrow if China wins at this. He was referring to AI. He said that on September 8th. So the United States views these stakes as incredibly important.

45:08 Now, what Dario is calling for is basically saying triple down on the export controls. We're not going to get China to agree to these safety measures. In fact, China has said in public statements that they're not going to slow down, no matter what America does. And so we have to slow them down. You know, we have to be the ones who ensure that the pace of safety technological progress is greater than the pace of generic overall AI capability progress.

45:45 And if China is not going to participate in that, then China needs to be removed from the technological frontier. And as bungled as the implementation of the export controls have been, they've still made a major dent in China's abilities overall in AI. I mean, American companies control more than 75% of the AI compute on planet Earth.

46:10 And very likely that number is going to go up by a lot next year, because most of the compute that's coming online is going towards American companies. And most of the compute that's coming online is vastly more capable than the compute that's out there right now. So the US has a huge advantage. If we were to restrict China's access to chips and mean it and enforce it, and we were to restrict China's access to cloud computing capabilities, essentially renting the chips abroad instead of buying them and importing them domestically, that would be a huge setback to China's AI capabilities.

46:50 Dario Amade estimates that it could widen the lead by as much as three to five years if this strategy was implemented correctly. I think that's plausible, although I would have had a much easier time saying that if we were talking about implementing it correctly in 2022, not in 2026. But the point here is, as Dario says, even a year to work on safety and pace the pace of frontier AI acceleration would do a ton for safety right now.

47:23 And if we can slow down China by a year, that would matter a lot. We could use that very, very productively. But because China is so close behind, if we are to spend a year on safety and they don't, it's totally plausible that they would pull ahead in frontier AI technology, potentially with an AI Chernobyl or an AI military takeover as the consequence for all of us.

47:50 It's a really tough time.

47:53 Greg, you mentioned that China had a response to Dario's proposal. Can we talk through what they said?

47:59 Yeah. So the foreign ministry spokesperson put out a statement at a news briefing. Here's what he said: quote, fear mongering, confrontation, and malicious competition will only disrupt the process of global AI governance and serve no one's interests. End quote. So then there was an op-ed in the Global Times, and the Global Times is Chinese state media.

48:23 So it's published by people who are paid by the Chinese government. Nothing goes out there without the approval of China's propaganda ministry. But as far as newspapers go, the Global Times tends to be a little bit more hot-blooded than, say, China Daily or Qinhu News Service. But here are some choice quotes from that Global Time editorial.

48:43 Quote: A close reading of Amade's proposal, however, reveals its hidden agenda. The article is packed with containment provisions targeting China and is, in essence, a Cold War playbook for the AI sector. Then it goes on to say: to prevent China from achieving geopolitical dominance by taking advantage of the U.S.

49:03 restraining its AI capabilities, he proposes choking off China to first widen the tech gap, then seeking conditional negotiations with Beijing to slow down. This includes three specific restrictions and containment measures: a strict ban on the sale of powerful AI chips or semiconductor manufacturing equipment to China, cracking down on so-called model distillation, and strengthening security at the AI companies and preventing model weight theft.

49:27 Then it goes on to say: this silent AI cold war is hypocritical and short-sighted. Its true purpose is to attempt to curb China's AI development through technological barriers and regulatory monopolies, uphold Washington's monopolistic hegemony and cutting-edge technology, and exclude China from the global AI governance system.

49:47 End quote. But that's kind of where we are, and that's kind of why using the export controls to slow China down is an attractive option.

50:08 So beyond China's statement, can we talk a little bit about how this landed more broadly? This story was front page, above the fold, in every major newspaper, really in the Western world over the weekend. So those in government as well as those in industry had plenty of reactions to it within about a day, and we've talked a little bit about this.

50:26 Sam Altman and Elon Musk and Demis Hasibus all publicly agreed with Dario in some way or another. What did everybody commit to individually?

50:35 Yeah, so Elon Musk just posted on X, quote, Dario is right, end quote. That's different from what Sam Altman said, right? Sam Altman said, independent evaluators are moving into my company soon. We're going to implement that same idea. Musk did not make any such commitment. Perhaps he will in the not too distant future.

50:56 Demis Hasibis over at Google DeepMind, I guess now he's the chair, no longer the CEO, but he posted, quote, Dario's essay points the right path forward. The details need working through, but the direction is correct for meeting this critical moment. End quote. Worth pointing out here two things. Number one, Hasibus had published his own regulatory model modeled on FINRA, which governs the finance industry.

51:23 And also, Hasibus was really the one poking former British Prime Minister Rishi Sunak to put together what became the AI Safety Summit in the UK in November 2023. So Hasibus takes the risks of AI super seriously. He's clearly on board with at least the direction of Amade's comments, if not the specifics.

51:49 And again, you have the four biggest titans of the industry all supporting this proposal. If there is an odd person out, it's probably Mark Zuckerberg over at Meta. But in terms of industry, I think this is a big deal that so many people are now getting behind this idea of pacing research based on safety progress.

52:15 It's still going to be tough to implement this. I remember, for example, the mood in the AI conversation when that AI safety summit happened. I remember, you know, what it was like, not when that first pause letter was proposed and pretty badly received in, I think it was spring of 2023, but the second one-sentence open letter that the existential risks of AI should be treated as a global priority, similar to nuclear weapons and global pandemics.

52:48 That seemed like it was really the crescendo moment and there was absolutely going to be some kind of AI governance structure. And then much of it dissolved. And the French AI Action Summit basically said that they wanted to undo a lot of the negative framing that the UK AI Safety Summit had put in place.

53:08 And the Trump administration said they wanted to undo a lot of the safety-minded thinking in the Biden administration AI executive order. So we've had moments when it seems like the AI safety narrative was coming together and reaching some kind of crescendo. I think this time is different than last time.

53:28 And I think it's different because of the OpenAI hugging face incident, principally. There's also whatever's going on in the classified realm. You know, we still in the public don't know what the Trump administration saw in the mythos model testing that so concerned them that they were willing to ban the mythos model worldwide using export control authority.

53:55 But that is, I think, the key thing here is that these hypotheses about how AI could be misaligned, these hypotheses about how AI could seek additional power and cooperate in pursuit of goals that are against the wishes and intentions of their developers, of their owners. A lot of this was hypothetical back in 2023.

54:21 You know, we had these early experiments that gave hints that this sort of thing was coming. Now we have real-world existence proofs of just how powerful these capabilities are and just how bad they can behave. And so I do think this time is different, but politicians kind of have to agree.

54:41 Yeah, and let's kind of hammer down on that one. So we talked a little bit about the industry side, and you mentioned a couple of governments and their reactions. Let's talk specifically about the Trump administration and their evolving stance on AI safety. You mentioned it last week. How does this letter and the reactions to it affect the characterizations that you made last week?

55:02 Well, you know, it's an emergency podcast. So here we are recording on the afternoon of September 14th. And President Trump has been busy today posting three times, at least as of this recording, on Truth Social on this. And normally I would not be one to long read President Trump's posts, but this one kind of merits the long read.

55:27 So here we go. I'm not quite sure how to represent the all caps in this post, but just trust me, dear listener, there's a lot of all caps. Okay, from the president, quote, the only controls or guardrails that AI needs is a strong and smart high IQ president. And the USA has that in spades. The Trump administration has stopped AI, quote, people, end quote, from doing bad or potentially bad, quote, things, end quote, like Dario Anthropic, who is now pretending to be a perfect little angel.

56:07 And we will continue to do so. We already have tremendous criminal and regulatory power over these companies. There is a sick conspiracy going on against AI and data centers, and the only one that is happy about it is China. Whoever wins AI wins. We are leading China and all others and will continue to do so.

56:28 Conspiracy theorists, treasonists, traitors, and leakers, beware. Thank you for your attention to this matter, President Donald J. Trump. Now, boy, there's a lot in there. One thing that I want to point out is just how up and down the company Anthropic and its executive, Dario Amade, has been in the mind of the Trump administration.

56:53 I think folks will remember that Trump was bashing Anthropic, culminating in Anthropic being banned from working with the Department of War, being declared a supply chain risk, although that ban is not being enforced at the classified level, at least not yet. Then there was something of a redemption arc in relations between the Trump administration and Anthropic when they brought Mythos forward and provided some very compelling demos to the administration.

57:23 And Trump actually said some nice things about Dario and how his behavior had improved. Then Fable V was released and shortly thereafter banned by the Trump administration. And Anthropic again fell in the mind of the administration and in Trump. And then finally they got their way back. And Howard Lutnick, the Secretary of Commerce, said earlier this month, quote, Anthropic is back on the right side with the Trump administration.

57:51 That was just earlier this month. And then this drops from President Trump. So surely it doesn't seem like they're back on the right side, at least not this week. That let's get into the substance of what he's saying. That number one, the government already has a lot of power over these companies, over these models.

58:12 In a sense, that's true. I mean, he banned Fable V worldwide by reaching for export control authorities. And there's the existing criminal and regulatory penalties. But that's not really what we're getting at here. A lot of what Dario is talking about is giving capabilities and capacity to these companies and incentivizing the right behavior and allowing for certain types of collaboration on safety technology that wouldn't be possible.

58:43 So that brings us to the second tweet. That was the 958 AM tweet. Here's the 133. Concerning AI, when in the history of business did anyone see the leaders of an industry call for regulation that, if strongly implemented, will drive them into oblivion and bankruptcy? AI taking over the world, destroying humanity, and all other things bad is a hoax.

59:06 No different from Russia, Russia, Russia, Ukraine, Ukraine, Ukraine, impeachment hoax number one, impeachment hoax number two, and all of the other hoaxes and scams that America was forced to endure through the destructionists and deviants, foul play, and illegal conduct. President Xi of China just announced that China will be doing absolutely nothing to stand in the way of AI or its future.

59:29 Google has recently stated that they want to build a massive plant in Finland, all because they are finding permitting too difficult in the United States. I am not happy about this and want them to change their thinking. AI and data centers will be the greatest economic development engine in history, bigger than oil, gold, diamonds, or even the internet.

59:47 It will not be stopped by brilliantly run destructive forces during the term of President Donald J. Trump. So I think the big thing in there is the note about China. So I want to be charitable to the Trump administration. Think about arms control negotiations. When the United States wanted the Soviet Union to build fewer nuclear weapons, they didn't lead in the negotiations with we're going to build fewer nuclear weapons.

1:00:24 What they said was our behavior depends on your behavior. If you are going to build a lot more nuclear weapons, then we're going to build even more nuclear weapons than you're going to build. However, if you were willing to build fewer nuclear weapons, or perhaps even reduce your nuclear arsenals, then perhaps we would also reduce our nuclear arsenals.

1:00:50 So Donald Trump as a negotiator, I can imagine him being somewhat frustrated with Dario Amade because he's about to go negotiate with Xi Jinping. And his negotiating leverage is not we're going to speed up full throttle. But if you were to slow down, maybe we would also consider slowing down. Instead, the negotiating position that the companies have left him with is, hey, my companies are about to slow down.

1:01:24 Would you please consider slowing down too? And so I think reading between the lines, that actually might be a lot of the source of Donald Trump's frustration. You can't say that he doesn't care about safety at all. If that's true, he wouldn't have banned Mythos. But remember, cabinet officials in the Trump administration in the past months have suddenly changed their rhetoric to some pretty pro-safety stuff.

1:01:54 Scott Bessant, who is playing an outsized role in AI policy, said that the dual goals, the two goals of the administration are maximum innovation and maximum safety. He's putting them in the same league. Michael Kratios, as I mentioned last week at the G20, was saying really nice things about the Center for AI Standards and Innovation, which he's now calling again America's Safety and Security Institute.

1:02:24 So the Trump administration has been making moves on safety. And if I had to understand what's the most charitable interpretation of President Trump's behavior here, it's gosh dang it, why couldn't you guys wait until after I met with President Xi Jinping to make these kinds of statements? So that's the that's the charitable interpretation.

1:02:51 The final tweet, I guess I read them out of order, but this one's from 1123 a.m. Quote, the only reason the AI slash data center outburst is happening is because the United States is leading by a lot, every other country. Don't kill the golden goose, President Donald J. Trump. And there, I think, is the final key to understanding Donald Trump's thinking on this topic.

1:03:15 There's plenty of economic analysis out there that says if the AI stimulus wasn't going on right now, the United States might already be in a recession, which is to say, a lot of economic sectors in the United States are contracting, but everything that's tied to AI is growing by so much that it's enough to offset all the other sectors that are shrinking.

1:03:39 So if AI progress is about to slow down, if the pace of acceleration of AI progress is about to slow down, does that mean that the companies are making a mistake by investing a trillion dollars of CapEx this year? Well, they evidently think not, but there were some concerning signals from the stock market this morning.

1:04:01 And you could understand how somebody whispering in President Trump's ear is that, hey, the leading lights of your technology industry appear about to commit suicide by slowing down the pace of improvement and jeopardizing the trillions of dollars of investment that are, in Trump's words, the golden goose.

1:04:22 So I really do think that those three keys help you understand where he's coming from. Number one, there is some appreciation and perhaps a growing appreciation of the safety issue, especially among the Trump administration's leading officials on AI policy, but arguably even within Trump himself. But there's the China threat and there's the economic threat.

1:04:49 And if you want to persuade Trump, you've got to explain how you're going to address those two concerns. That's at least my thinking.

1:04:58 So, Greg, let's talk about a couple of more perspectives here. There certainly was a mention of President Obama and some remarks that he made in the coverage this week. And also, if you're talking about AI with the Trump White House, you kind of have to at least take some consideration of where David Sachs is on this issue.

1:05:15 So can we talk about both of them briefly?

1:05:18 Yes. And I thought it was noteworthy that Obama not only said something, but it got picked up by a lot of media outlets as significant. So he last week was speaking at a Democratic Congressional Campaign Committee fundraiser, which is the former president's opportunity to provide advice to what the Democratic Party should be campaigning on for these upcoming midterm elections.

1:05:43 And he said that he was not an accelerationist and not a doomer, but quote, if I were running for president next year, this would be one of my central agendas. And I would have a very clear plan and I would talk about this and I would say, here's our plan for safety. Here's our plan for making sure our kids are not corrupted by this.

1:06:03 And then he goes on to say, I would be thinking about the economic impacts in very concrete ways and understanding what does it mean if there's going to be job displacement. End quote. And then he went on to say in a discussion with uh House Minority Leader Hakeem Jeffries that Jeffrey should convene a public conversation.

1:06:24 Quote, once you are speaker, I would strongly urge that the Democrats put together a framework for a very public conversation. And you can break it out into three or four or five issues and lay the foundation for a broad public conversation, Obama said. So that's pretty remarkable. I think if you look at AI as an election issue, we've seen, as we talked about in an earlier episode, Republicans view AI data center unpopularity as potentially putting their congressional majority at risk, especially in the contentious Senate race in Ohio.

1:07:00 And here you have Obama saying this should be a central part of the Democratic Party's agenda. Now, I don't think there's a cohesive picture in the Democratic Party right now in terms of what AI policy should look like. But you've got a former president and a current president loudly talking about these issues, which, as you said, Adam, were on the front page of just about every major newspaper in the country this weekend.

1:07:28 Yeah. So very briefly, let's talk about David Sachs and if he's positioning on this at all publicly right now.

1:07:35 So David Sachs, you know, as we mentioned, has attacked Anthropic, has attacked Dario's essay. And he did say the risks may merit slowing. And it probably was sarcastic, but he basically said, quote, if the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible.

1:07:58 Then he goes on to say, quote, but stop pretending you need anyone else's permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need regulatory approval process that supersedes product liability. Stop pretending that you need those same evaluators to police competitors who aren't even at the frontier.

1:08:20 Stop pretending the motivation to slow down is purely altruistic. You face massive product liability exposure if your products enable a truly damaging cyber attack. The market already punishes models that behave in unpredictable or unauthorized ways. It is just also giving customers what they want. And then he finally said that whatever thing that you come up with, China's not going to join.

1:08:44 So fair to say that he is number one opposed to almost everything that was said there, is still making similar arguments about small AI or what the industry likes to call little AI, an idea that is being pushed by A16Z and others. And he clearly still has the ear of President Trump on this issue. And so he is a force to be reckoned with.

1:09:11 You know, he's getting in fights on Twitter with former Trump AI policy senior advisor Dean Ball, who's now at OpenAI. He's attacking Dario quite openly. But I think there's movement on these issues, and ultimately, this is going to be a key election issue in 2028. And no matter how loudly David Sachs yells, Republican-elected officials are going to be listening to the polls.

1:09:39 All right, Adam, you're on Capitol Hill just about every week. So let me flip it around here and ask you the question: what do you think is going on in Congress? And maybe let's start with uh House Republicans.

1:09:52 Yeah, Greg. So Speaker Johnson had a reaction that I think that we could list as relatively predictable. He very much uh in line with David Sachs, put the onus on AI companies and resisted immediate congressional action. He also realistically just wants to get his people out of town, right? He wants to get all of his members out to the campaign trail.

1:10:12 So this wasn't something that he was going to take weeks, if not months, prior to the campaign to focus on. So the quote from him directly is there is an obvious corporate responsibility that the people who are creating these models have to ensure that their products are safe. That was a statement that he made to CNN State of the Union, their Sunday show.

1:10:33 We have to resist Congress jumping in and imposing some sort of emergency moratorium. It's got to be done the right way, safely and wisely. End quote. So E doesn't want any of this right now. He's trying to stay as far away from it as humanly possible. If you read that for what it is, that is less a concern about the subject matter itself.

1:10:51 He is very, very focused on trying to fight back against a perceived blue wave in the upcoming election. He already removed time from the congressional calendar to send his members back out to the campaign trail faster. He just wants everybody out of town, right?

1:11:08 So I think he broadly speaks for many House Republicans, although there's probably a pretty significant diversity of views on this issue, especially as it relates to child safety. But let's now turn to Democrats. There was a bunch of letters flying around all week. What do you see happening?

1:11:24 So uh there was a letter in particular from four House Democrats. It came from Representatives Sam Licardo, George Whitesides, Lori Trahan, and Ted Lew on September 11th, asking Speaker Johnson to bring the House back into session immediately and remain in session until Congress advances meaningful bipartisan AI safety rails.

1:11:43 And the letter specifically cited mass cybersecurity breaches and development of biological or chemical weapons as the threat. That letter's interesting for a number of reasons. When you look at the authors of the letter, and I typically kind of focus on the people themselves, Sam Licardo represents a significant chunk of Silicon Valley.

1:12:00 He kind of took uh Anna Eschew's old seat. So he represents Silicon Valley. George Whitesides is kind of out towards Lancaster in Southern California, but he is a former NASA chief of staff. And obviously, you and I know him from our space days. Uh he was a virgin executive at the time. Most importantly, out of almost anybody, is Lori Trahan.

1:12:19 So Lori Trahan's a member from Massachusetts. She's younger than your average member and very focused on this policy issue. And she is foregoing certain career opportunities at this point in order to stay focused on this and attempt to become the leader in AI policy. So I think that that, along with Ted Lu, who's a member from Los Angeles, kind of represents some of the areas in coastal Los Angeles, would give you a pretty solid focus on where the actually engaged people in the Democratic Party are on this issue.

1:12:51 I mean, these are at least four of, if not the four, self-styled AI leaders in the House, Democratic side, right?

1:12:58 That's absolutely correct. So it is marked that the four of them kind of came together and put this letter forward that said nothing is going to happen in Congress before the midterm elections. Everybody's out of town. The larger concerns really have been government funding through the end of the year.

1:13:13 So the CR has been passed trying to figure out how to get a full appropriations package together for FY27, let alone the defense appropriations work that's been going on between the various proposals that are out there and money to backfill the conflict with Iran is much more top of mind for members right now, although this is certainly creeping up.

1:13:33 We've seen the minority leader and presumably future Speaker of the House, Hakeem Jeffries, pressure Speaker Johnson on this issue. He'd really like to see the House return and spend time working on this issue. But as the New York Times has rightfully said, one week is a very short timeline that makes it nearly impossible for lawmakers to quickly adopt a comprehensive framework for AI regulations, and they are not wrong.

1:13:57 That is a very accurate statement. So those four members and the speaker are certainly not the only members of Congress who have been in this space and working in this space. Greg, I know you've been paying attention to some of the work that Senator Bernie Sanders and also Representative Greg Cesar have been putting in.

1:14:13 Can you talk a little bit about what they're looking for, what their policy push has been?

1:14:18 Yeah, so here I have to just provide some little bit of context. You know, when I was in the Department of Defense, some leading AI companies, like their executives or their government affairs folks, or their engineers, would tell me that they had real concerns about artificial general intelligence and artificial superintelligence, but it was kind of whispered in hushed tones.

1:14:42 Because at the time in Washington, DC, saying that you had concerns about what AI was going to do when it was smarter than Albert Einstein basically got you discredited in most polite policy conversations. Well, Bernie Sanders, a senator, a former presidential candidate, has now introduced the Ban Artificial Superintelligence Act.

1:15:10 And according to his press release, that act, quote, would permanently ban the development and deployment of superintelligent AI and temporarily pause advanced AI development until a federal regulator has established safety rules. It would also direct the United States to pursue international agreements to prevent superintelligence from being developed anywhere in the world.

1:15:32 Now, I think most people know this, but just for those of you who do not, Senator Sanders is among the leftmost senators in the Democratic Party in terms of political ideology. And he also has a history of introducing bills as what you might call the opening gambit of a negotiation in which he's trying to anchor very high in terms of what he wants to expand the Overton window of what might be possible in the course of political negotiations.

1:16:07 So back in 2024, Fize Shakir, who was Fernie's former campaign manager, said that one of the things that they would do is push for stuff like Medicare for All. And it's not because Senator Sanders thought that he had a shot of getting Medicare for All passed through the U.S. political system, but it was sort of the opening negotiation position with the hopes of getting stuff like maybe they could lower the Medicare age from 65 to 60, that sort of thing.

1:16:39 So I'm reading between the lines here, but I suspect that's also what's going on here with AI. Sanders is out there proposing banning superintelligence. He's out there proposing banning data center construction in the United States. I sincerely doubt that's what he expects is going to pass in Congress.

1:16:58 It may not even be what he wants, but what he does want is serious attention devoted to the issue of AI safety. And by introducing bills such as these, ban superintelligent AI, he's kind of telling the entire country, hey, you should talk about this. And this situation is so crazy right now that people are talking about banning it.

1:17:24 And as an opening frame for the conversation, he's probably succeeding in terms of moving the Overton window of the country. And I suspect that's what he's trying to do.

1:17:34 Yeah. Senator Sanders certainly is laid down, as you mentioned, a number of different marker bills in the past, which is kind of the term of art for these pieces of legislation. They're not fully fleshed out. They're not in a place where they could be passed in the way that they are, but he absolutely is kind of going with the zeitgeist at the moment and focusing on this issue, especially since it's such a big issue in the midterms right now, at least as far as data farms are concerned.

1:17:57 Totally, totally. That'll do it for another episode of the AI Power Podcast, though I'm sure that there will be plenty more to follow up on in this space, not just on the podcast, but in the national conversation. We've had an entire episode planned previewing President Trump's upcoming talks with Chairman Xi Xin Ping, which we'll hope to bring to our listeners later this week.

1:18:16 Our thanks go out to our audience for tuning into this emergency pod, and as always, to Greg. We'll be back soon with another episode, and as always, I'm sure we'll have plenty to talk about.

1:18:25 Thanks, Greg. Thank you, Adam, and thanks, Jack Conlin, for producing this episode.