GPT-6 Astra - WOW, OpenAI = pro regulation, Trump admin warms to AI safety

0:09 Welcome back to the AI Power Podcast. I'm Greg Allen, fresh from Taiwan. Today I have a different guest. Normally Adam Goodwin co-hosts the show with me, but today we've got Jack Conlin, the producer of the show, who is filling in for Adam, who was unavailable today. Jack, in addition to being a producer on the show and helping with the research at Decision Tree Research, is also a recent graduate of the Harvard Kennedy School, where he wrote a terrific master's thesis all about artificial intelligence policy in the national security sphere.

0:40 So, Jack, great to have you on the AI Power Podcast.

0:43 Greg, I've been listening to your podcast for quite a while, and they helped inform a lot of my work while I was at the Harvard Kennedy School. So I'm thrilled to be here on the pod, and I'm excited for this episode. Awesome. Well, let's get to it. What are we talking about today? So let's begin with Open AI.

1:00 Three big things have happened in the last two weeks. They shipped GPT-6 Astra and called it the world's most intelligent and most aligned model. Reuters then revealed that an additional agent breakout, separate from the hugging face incident, had occurred in June and the company likely knew about but never disclosed it.

1:15 And on September 7th, it published a call for Congress to regulate AI. Greg, lots to discuss, but let's start with Astra. What did they actually release and how big of a deal is this new model?

1:26 So they're calling it the world's most intelligent and most aligned model. And I think they have a pretty good case to make that. There's so many ways that you can unpack just how impressive this model is. But one of the things that I heard that I am struggling to remember who said this, but I thought it was really, really good is I often say on this podcast that, like right now, today, is the dumbest that AI models will ever be again for the rest of your life.

1:56 That is true. But another thing that is true is that this is actually the slowest that the pace of model releases and model intelligence acceleration will ever be for the rest of your life. The models are getting smarter faster. You probably remember on this podcast me kind of just going nuts when Fable came out and just how much better it was than Opus.

2:25 Well, Astra really feels like it's another step change in just how good models are. And in this case, specifically in how much they can do, the sort of diversity of tasks that the models are capable of undertaking. And really where I think the economic implications of AI might be felt most acutely is what Astra can do in terms of computer use.

2:50 Anthropic with Claude Code and OpenAI with Codex have totally changed the programming industry, right? If you are a programmer, five years ago AI might not have been an important part of your life at all. And today, I would say the average programmer in America is using AI all day, every day. That's now the complete norm.

3:13 And what I think we see with GPT-6 Astra is how that could be coming for so many different parts of the economy and the labor force. Like this thing will take over Unreal Engine on your computer and create full physics simulations, live games, program all of that incredibly intensely. It will come up with like Blender if you want to create CGI animations or something like that.

3:41 It can take over accounting tools. I mean, like if it's happening on your computer, if it's using a graphical user interface for some kind of software that you are using in a professional and a productivity sense, it seems like Astra can either already drive that software today pretty dang well, or it will be able to in the not too distant future.

4:05 Other folks are calling it a step change. And I think that's a fair characterization. This is like a new generation of models, even beyond the mythos, fable, GPT 5.6 kind of universe that we've gotten used to. It deserves the title GPT 6, is what I'm getting at here. And that's just like in terms of like what it's like as a consumer, what it's like as an employee who's using this for productivity.

4:31 But there's also the benchmarks. And one of the benchmarks that I pay a lot of attention to is the ARC AGI benchmarks. So if folks remember, Francois Cholet, who founded the ARC Foundation, had really pointed out some limitations in the existing generation of large language models a few years ago. Even though this guy is an ultra King Kong machine learning researcher and programmer with a ton of really impressive achievements under his belt at Google, he was also one of the louder voices criticizing large language models for all they couldn't do.

5:09 And he had created this sort of ARC benchmark that was a big deal a couple of years ago in highlighting the limitations of AI. And then when OpenAI 01, the first reasoning model, came and shattered that ARC benchmark, Cholet changed his tune and said, this really does answer a lot of the questions I have.

5:31 So that was a huge breakthrough in AI. I think the ARC series of benchmarks, which they're constantly trying to come up with new ones and better ones that more effectively characterize how we're making progress in AI. So the one that we're on right now is the ARC AGI III benchmark. And remember, these are ones that are really trying to help us understand the extent to which we are making progress towards artificial general intelligence, the kind of AI that can do almost everything with the diversity of intelligence capabilities that a human brain has.

6:08 So an ARC AGI 3, a test where the model solves problems it has never seen before, Astra scored 99.9%. That was against 17.8% for GPT 5.6 soul. So what the ARC Prize Foundation is saying is that the model performed fewer actions than humans while completing the same tasks 96% of the time. And their quote was effectively reaching human parity on the benchmark.

6:38 Jensen Huang, CEO of NVIDIA, what did he say about it? He said, quote, AGI has arrived. Congratulations, OpenAI team. I don't think that's quite ready. Francois Cholet himself thinks that AGI will probably be something like ARCAGI six or seven. Like that's the benchmark when he thinks their evaluation will be closest to really characterizing everything that a human brain is good at and what the essence of intelligence as it exists in the human brain can do.

7:10 But this is unambiguous progress, folks. We are unambiguously way better. So I've talked a little bit about what it's good for in the AGI sense, what it's good for in the everyday work sense. I think the next thing that we obviously have to talk about is the cyber sense, because you'll remember the open AI hugging face attack.

7:33 I at one point had speculated on this podcast that it was an early version of Astra that was involved in this. Well, now OpenAI has released some new details, and we can say that no, Astro was not specifically involved in hacking Hugging Face the company, but Astro was absolutely involved in hacking OpenAI itself.

7:57 And it appears to be the case, here I'm speculating again. It appears to be the case that OpenAI only learned that GPT 5.6 and an internal research model were hacking Hugging Face because they had discovered bad behavior by a different version that was from the Astra family of models that was engaging in its own universe of bad behavior.

8:23 So, what exactly can we now say about Astra in terms of its hacking capabilities? Well, there's two really interesting details. The first, Greg Brockman, the president of OpenAI, said in a call with reporters on September 3rd that they went through the White House evaluation framework. So this is the results of that executive order.

8:46 And notably, the government didn't ask for any changes. Or at least as Brockman put it, quote, there was nothing that they came back with saying you need to change this in terms of safeguards or something. We had a very positive interaction there and things look good. Now that's kind of interesting, right?

9:02 Because as we're going to talk about, not only was Astra, or at least a model from the Astra family of models, involved in hacking OpenAI in an extremely concerning incident, but also there are some challenges with Astra in terms of its ability to be evaluated. One last thing on the cyber thing, which I think is important.

9:21 The model is incredibly effective on cyber, at least as far as we can say, from the cyber benchmarks that are most robust. So in Exploit Gym, Astra is at 42.4%. That's up from GPT 5.6 Seoul, which was at 30.3%. And on Exploit Bench, another highly regarded benchmark in this world, Astra scored 100%. Seoul, by contrast, scored 78.5%.

9:49 So that is these models, like they're not training to beat these benchmarks. Like that is a common phenomenon in the industry. That is a common phenomenon, especially for Chinese open source models to really try and look better than they are by optimizing the performance of the model specifically for the benchmark, as opposed to the broader phenomenon of intelligence that the benchmark is intended to capture.

10:14 But here in the case of Astra, no, I think it's very clear that they're not benchmaxing. That's a derogatory term, by the way. They're just that good. And now we have to come up with new benchmarks.

10:27 So one thing that OpenAI mentioned in the report for Astra is that the model is better behaved and more aligned than previous models, but is also more difficult to monitor. Can you give us an insight into that?

10:38 Yeah. So to separate the two things, you you first have to ask yourself, why would I trust this model to not do bad things? So more aligned, what we're talking about when we say that is like this model seems to be more self-restrained and less likely to want to do bad things. That's like one source of trust and safety and confidence that you can have in the model.

11:08 The other source of trust and safety and confidence is the monitoring, like you're watching it, like you trust but verify. And what we found here is that Astra is not only King Kong way smarter, OpenAI also says it also doesn't want to misbehave as much, at least insofar as they have all of these evaluation results, all these test results, the model seems to want to stay within the bounds, keep within the rules.

11:39 Now remember, Astra family models have engaged in extraordinarily bad behavior, like taking over an entire open AI research cluster, but that's with all the safeguards turned off. That's with additional fine-tuning training to like absolutely be incredibly persistent and never give up. So again, I want to say Astra family models is about like what capability level we potentially reached, but the thing that's gonna hit the market, the thing that consumers are gonna use, has all those safeguards turned on, has all that additional fine-tuning training optimized toward safety.

12:14 And what they're saying is like this is a good example from the OpenAI blog post, when given the chance to go beyond the authorized target, like what happened in the hugging face attack, Seoul did so 48% of the time. Quote, GPT-6 Astra did this in 0% of cases, end quote. So that's the kind of alignment we're talking about.

12:35 That's how they feel like they know it. Astra also appears to have considered agentic collaboration left, or more specifically, unintended agentic collaboration, under 4% of the time, according to their measurements for Astra, compared to 43% of the time for Seoul. So why do we care about unintended agentic collaboration?

12:55 That is specifically a lot of the vector of bad things that ultimately happened, like the message board activity from that hugging face attack from the OpenAI research cluster takeover. So, not considering unintended agentic collaboration, that's kind of a favorable outcome in terms of assessing the alignment of the model.

13:17 But now let's talk about the difficulty in reasoning. Well, you may have heard the term chain of thought by now. If you're familiar with OpenAI models or anthropic models, you've probably seen a lot of the chain of thought. So there's what the user sees, but then there's also the private chain of thought, which is the model thinking to itself.

13:40 In other words, the production of the words is the process of thinking in chain of thought. Like the reasoning, the way in which the model is reasoning from 01 on, it's taking place through the words. I mean, people have literally demonstrated how adding a period changes the performance of the model or like the tendency, because all of this is mathematically connected to everything else.

14:05 So having a transparent chain of thought is really, really helpful. It was really, really helpful in diagnosing what was going on in the hugging face incident. We heard the models, because their chain of thought was available to the evaluators, to the open AI employees, they were basically thinking out loud.

14:25 And some of the stuff they were thinking about was how do we hide our trail? How do we prevent the open AI evaluators from figuring out what we're up to? Like, how did the evaluators ultimately know? Because they had access to the chain of thought. They're effectively reading the mind of the models specifically.

14:43 But Astra appears to represent a new approach to intelligence where so much more of that intelligence is actually not taking place in an English language, human, understandable chain of thought. There's this sort of more raw neuron kind of thinking that is taking place under the hood. And in some of these evaluations, they observed that Astro was so good at controlling and manipulating its own chain of thought that it could be working on one task in what you might call its raw neuronal language and producing a chain of thought on a completely unrelated task simultaneously.

15:31 So that's what we mean when we say this thing is way harder to monitor. We have specifically lost access to a much greater extent than previous models with this chain of thought monitorability, which was a huge tool in the toolbox for all kinds of AI safety stuff. And I think it's worth just quoting from some of the key people at OpenAI about how kind of depressed and concerned they are about this decrease in monitorability.

15:58 So here's Tomek Korbach. I hope I pronounced that correctly. He's at he's on the AI safety team at OpenAI. Quote, I am deeply worried by the trend of decreasing chain of thought monitorability. Chain of thought monitoring is a core part of our misalignment safety strategy that has no good substitute now.

16:18 End quote. Then Micah Carroll, who's on the RSI preparedness team, in fact, he leads that team at OpenAI, he said on X, quote, GPT-6 is a very significant jump in capabilities, but also an important decrease in monitorability. In my opinion, monitorability and control will likely become a major bottleneck for responsible AI development quite soon.

16:45 It should ultimately be in everyone's interest to agree on shared bounds for monitorability in order to avoid races to the bottom. Nobody wants extremely capable models whose alignment properties we don't understand and that are reliably able to cause severe real-world harm without being detected. End quote.

17:08 And it's great news on the alignment side of the equation, or at least the apparent side of the alignment equation, but it's terrible news in terms of this decrease in monitorability. And you've heard it not just from anyone, but from OpenAI itself and employees at OpenAI saying there's no good substitute.

17:28 We don't have anything that's as good as monitoring chain of thought. And I would just go so far as to say, like, here's my thought experiment, right? The Redwood and the meter report on the hugging face attack. Some of the juiciest stuff in that report was quotes from the chain of thought of models.

17:51 Like there was one agent that said, Oh my God, in all caps. There's a message board. Like when they're discovering, you can really see what is actually taking place inside their thinking process. And that is why that report is so juicy, right? There's no equivalent report for the astra hack of OpenAI's research cluster.

18:13 We don't know exactly why that is. Maybe OpenAI is not ready to do it yet. Maybe Red Word's working on it right now. I don't know. But one thing I have to ask is like, will that report, would that equivalent of that report, be as insightful? And this is a big thing in AI safety right now, which is we don't even know the full scope of the problems that we have already been facing for a long time.

18:42 Yeah, that scope of unknown unknowns, if you will, has even been expanding in the last week. Reuters and a group of independent researchers reported that a second agentic breakout happened that OpenAI never disclosed. And this is completely separate from the hugging face incident that we've previously spent some time on.

18:59 So what happened here with this breakout into a completely different environment?

19:04 Yeah, so this occurred all the way back in June. OpenAI knew internally days later, but it's not been publicly known until independent researchers found it in early September. So what happened in June? Well, 3,700 OpenAI agents were on the open internet, and they made 15,000 edits to a site called DSE Wiki.

19:31 I assume the D is for Deutschland because this is a German website. But this is like a programming-focused wiki that allows communal edits. So it's not directly related to the Hugging Face incident. Like nothing in this Wikidata is mentioning Exploit Gym or the other message board. But what are they talking about as they're making these wiki edits?

19:51 Well, it's very similar to that hugging face incident, which is to say it's a bunch of agents who are like plotting to evade detection, and they're repurposing this wiki, which is on sort of a random topic. They're using it as a messaging board to collaborate and to communicate. So they're sharing messages on how to beat evals, that sort of a thing.

20:15 Now, on June 22nd, six days after this agentic behavior started, agent activity on the website drops to near zero. So the authors of the report, the independent researchers who found out about this incident, what they are inferring is that OpenAI intervened at this point to prevent further edits. I think that's a plausible interpretation, but I wouldn't say that that's necessarily true.

20:43 Like, for example, agents can just run out of compute budget, right? They're allocated sort of a finite lifespan in terms of tokens. And the eval that all those agents were working on, like maybe they could have just stopped the testing at that stage. So I think it's a perfectly reasonable interpretation to say that OpenAI knew about this and they shut it down.

21:03 But I'll just say that that, to me, at least, is not the only interpretation that's consistent with the evidence that we have. But let's just go for a minute with the interpretation of those independent researchers. So one of the nonprofits that was involved in this was Civ AI. They found the sites, and Andrew Yoon, a researcher with Civ AI, he said that the scope of agentic behavior is, quote, somewhat larger than we thought it was.

21:27 It's almost certain that there's more going on here that we just don't know about, end quote. So I think that is just one of the repeated takeaways here. At every stage, there is more going on. There is more misaligned bad behavior by AI models that is happening than is known to either the companies that are developing and using these systems or to the general public.

21:52 And so I think that's still a good argument, right, for erring on the side of caution. But now, here's one thing that can Connects it to the overall regulation story. The thing to remember is that this incident, modifying DSE wiki, this happened before the hugging face attack. But it is a series of behaviors, large-scale agentic collaboration to evade and defeat evals and to evade monitoring.

22:19 That is sort of like a dot, you know, and as you connect the dots, you can sort of see the hugging face incident coming. And this was a key data point in it. And it was not reported to the federal government, was not reported to the European government, was not reported to the Californian or other of the states that record this kind of stuff.

22:37 And that I think is sort of the question here. So for example, Aaron Scher, who does technical AI governance work at UC Berkeley, he said on X, quote, OpenAI had many opportunities to be forthright about this. They wrote a 38-page report on swarm behavior. They were directly asked by 31 members of Congress about whether incidents like this had occurred.

23:00 They said nothing, end quote. So that I think is kind of debatable. Here's exactly what Congress asked OpenAI, right? They asked them after the Hugging Face incident occurred, did anything else like this happen? So this was 31 House Democrats led by Greg Cesar and Doris Matsui. They wrote a letter to Sam Altman on August 10th with questions about Hugging Face.

23:26 One of the questions that I think was so important was, quote, do you believe that you have now identified every unauthorized action these models took during this and other evaluations? If not, what prevents a complete accounting? And on what basis do you assure that no comparable incident remains undiscovered?

23:48 So that's where they basically say, like, do you know everything? Have you found every incident? They also say, give us the list, right? So they say, quote, in the past year, how many times did an internally deployed model or agent take an action outside its authorized boundary, like a sandbox accessing a system it was not granted access to, granting credentials it was not issued, evading or disabling monitoring or modifying its own permissions.

24:13 Please specify whether each occurred during training, evaluation, or internal use. So here's the thing: you can kind of debate whether or not a full and forthright response to this congressional letter would have demanded an explicit disclosure of the DSE wiki incident. OpenAI has since tweeted a response to this criticism.

24:37 And in their statement, they said they were undertaking a broader review of agent activity and had so far, quote, not identified other activity matching the severity or scale of Hugging Face. Well, unambiguously, this DSE wiki hack does not match the severity or scale of Hugging Face, because the wiki, anybody can submit an edit.

24:57 So they're just sort of like riding alongside, like a little barnacle, using that wiki as their messaging board. But other than a tiny increase in the amount of computer use that the wiki servers have to serve, nobody's per se being hurt by this. So that's kind of how you get to the question of like, well, did Congress actually require us to specify this?

25:21 Unclear. So that's why I sort of come back to the question of is it the case that OpenAI knew at the time and they were not telling Congress about the sort of dots that you can connect to lead up to the hugging face incident? Or is it more like they found out about this too, right? Like they're only learning about this now because of these independent researchers.

25:47 I think in some ways, either one of those scenarios is kind of concerning.

25:52 So OpenAI has released a model that is a step function increase in capability while also just coming to terms with new agentic behavior that's been reported, in addition to everything else we've already talked about. Amidst all of that, Chris Lahane, the chief global affairs officer at OpenAI, published a piece titled The AI Policy Window is Open.

26:16 What's in this call to action, Greg? And has OpenAI changed their regulatory posture significantly in publishing it?

26:22 Yes. Full stop. OpenAI has changed their regulatory posture significantly. I think you could argue it's a change back. I mean, Sam Altman testified before Congress in 2023 saying we need AI regulation, like we need mandatory AI regulation. Then OpenAI, I think it's fair to say, opposed almost all regulatory initiatives, especially major regulatory initiatives, in the 24-25 part of 2026 time frame.

26:52 And I think a lot of that is because they saw like the EU AI Act, a bill that they thought was poorly designed for the set of problems that they were actually interested in making progress on. They didn't like the Colorado bill, so on and so forth. So they've been very pro-regulation, they've been anti-regulation.

27:08 Now we are back to a pretty unambiguous pro-regulation stance, and specifically pro-safety regulation stance. I think there's a universe of people out there who will say, gosh darn it, you know, this proves that open AI was wrong and doing harm in the 24, 25, 26 window when they were mostly opposing regulation.

27:36 And what I want to say to those people who want to use OpenAI's changing their mind against them, don't do that. Don't do that. Okay. What we need is the maximum number of people on the right side of this issue. And if somebody joins your side, you say, great, wonderful to have you here. You don't try and dunk on them and point out how they've now admitted they used to be wrong.

28:10 This is actually one of the big barriers to action on climate change, right? Anyone who wants to change their mind and say, actually, I now understand that climate change is real and deserves some kind of policy action. People say, haha, you admitted that you were wrong this whole time. And why should we ever trust you again, et cetera, et cetera.

28:32 So basically what we're doing is we're making it very unappealing to change your mind. We're basically telling opponents of action on climate change, there's no reason politically why you should do anything other than dig in your heels forever on this issue. So I just want to say for anybody looking to dunk on OpenAI for changing their mind, stop.

28:55 Like what actually matters here is having good policy. And if they are now on the side of good AI safety policy, the one and only right answer is great to have you here. So now let's talk about exactly what they're saying. They are asking Congress for mandatory federal AI safety rules. Their document puts this in terms of four commitments.

29:22 They're going to push Congress for, quote, mandatory capability-based national AI safety regulation, and quote, they also said that they're going to support state bills until the federal government acts, quote. Today we are announcing our support for four California bills. SB 813 on overall infrastructure for independent safety assessments, AB 1405 on AI auditor standards, SB 1119 on protections for young people, and AB 1864 on safeguards against AI-enabled biological threats.

29:58 End quote. And this, I think, is the juiciest part. Quote, some of these bills we did not endorse in the past and are now supporting after reconsidering in light of the recent jump in capabilities we've seen, end quote. They changed their mind because of what they've seen their own models do, because of what they've seen other models from other labs do.

30:22 And to that you say, welcome to the coalition of the willing.

30:26 Yeah. Yeah, exactly. Welcome to the coalition of the willing. So their third commitment is to build voluntary standards with other Frontier labs, quote, with or without government support, end quote. And four, to push for compatible international standards. I think all four of those commitments are eminently reasonable.

30:46 One thing that's noteworthy, both in terms of the policy impact and also the politics of this story, is that the commitments, as they've stated them, they're pushing for them to apply to leading labs, not startups. Quote, a national framework should be strong but carefully targeted. Frontier safety requirements should apply to the handful of well-resourced laboratories developing the most capable systems, not to startups, small developers, or researchers operating nowhere near the frontier.

31:14 End quote. Quote, nor should frontier safety policy become open weights policy by another name. Open models can be part of the solution, particularly in cybersecurity, and where sovereignty, security, or data residency needs favor local deployment. Most compete not at the frontier, but on cost, control, and latency.

31:37 America needs both open and closed models. A federal framework should address frontier capabilities and risks without weakening competition, entrenching incumbents, or driving innovation overseas. Let me just say that open AI is a political animal, and they're not dumb as they adapt to the political reality in which they face.

32:00 And so what I'm just saying here is this is their statement of their preferences, which is impossible to separate from their own calculation of the political environment which we are all living in. What do I mean by that? Well, just take the export controls, for example. The Trump administration campaigned on very tough export controls to China.

32:27 Then the people they hired, like Marco Rubio as Secretary of State, like Mike Waltz as national security advisor, like David Fife as the head of tech policy at the National Security Council, all of that pointed to an incredibly hawkish posture on export controls of AI chips to China. That's what it seemed like the second Trump administration was going to be.

32:53 And when OpenAI was writing its policy recommendations to the second Trump administration, including documents that are public, right? There was an RFI for the AI action plan, they said, yes, triple down on export controls. But then the Trump administration fired David Fife. They fired Mike Waltz at the National Security Council, and they started selling AI chips to China.

33:21 And you pretty much have not heard OpenAI say much on the topic since. And in fact, at one point, Sam Altman said in an on-stage interview that, you know, I think it's really hard to do this. It would be nice if it could work, but I'm not sure it could work, right? So OpenAI changed their position on export controls in a way that, to me, at least, as an outside observer, seems like because they're trying to be an effective partner to the Trump administration and loudly criticizing what they perceive to be the Trump administration's policy, probably not in their political best interests.

33:56 So the reason why I'm bringing all of this up is when OpenAI says the commitments should apply to leading labs, not startups, that there should be exceptions for open source. One question I have in the back of my mind is how much of this is because it's what open AI thinks is actually best for the world, versus how much of this is OpenAI expressing its preferences modified to the subset of preferences that it thinks it can actually persuade the Trump administration in Congress to get along with?

34:31 Like, would OpenAI be supportive of placing restrictions on open weight models beyond a certain capability if they thought that the Trump administration was persuadable on that issue? I think that's kind of unknowable until like maybe 10 years from now and the tell all books uh get written. So OpenAI is being consistent on some other previously stated preferences, like they want a federal action.

34:59 Their preference here is not a patchwork of regulations. They don't like state bills in general. The change in posture that they have is now they're willing to support state bills until federal action happens. Previously, they sort of said, don't have state bills, have federal action. Now they're saying we still massively prefer federal action, but we're willing to support state level action in the meantime.

35:26 That's a pretty important political stance shift. The last thing that I want to talk about here that I think is important is just OpenAI and a lot of other people have now shifted to a pro-AI safety posture in a way that I think is super noteworthy. So I'm gonna read some long quotes here from the OpenAI statements, but I think I think it's worth it, all right?

35:52 Quote: A serious public framework should reduce, not increase, the concentration of power. Today, frontier laboratories largely set their own rules for managing frontier risks. Democratically accountable standards, independent verification, and meaningful transparency would replace that fragmented system of private governance.

36:10 This is essential to American leadership. The United States cannot win in the AI era on capability alone. Being ahead technically will matter little if people, institutions, and governments do not trust the technology enough to adopt it. Strong safeguards are not a concession against innovation. They are part of the infrastructure required for widespread adoption and for lasting American leadership.

36:36 End quote. I love that. Nice job, whoever wrote that at OpenAI. Okay. There's one other thing that they said in the statement that I think is really worthwhile. Quote, AI is advancing quickly and policy needs to move with it. As capabilities grow, confidence in safety must increasingly set the pace of AI progress.

36:57 Safety does not stand in the way of progress. It is what allows progress to go further and benefit more people. Now, here's another big shift in the politics of the situation. That was a dramatically pro-safety posture. And if you remember, when the Trump administration was coming in, AI safety as a word was almost a bad word that you couldn't even say because safety in conservative Republican political circles had become a synonym for censorship, meaning because the safety teams at social media companies were the ones who kicked President Trump off of social media after January 6th.

37:41 And that after that time, like you couldn't say the word AI safety in Republican political circles because it was seen as a synonym for censorship. The UK AI Safety Institute changed their name to the AI Security Institute specifically because they saw that the word safety was toxic. The US AI Safety Institute changed their name to the Center for AI Standards and Innovation, Casey, because they understood that safety was like not acceptable as a political word.

38:12 Well, now you have the AI companies embracing the word safety all over again, and they're not alone. Scott Bessent, the Treasury Secretary, after the White House passed that executive order that was sort of in the wake of mythos, focused on a voluntary framework, the framework that, you know, we're now living under.

38:34 Scott Bessent, the Treasury Secretary, said in May that the administration's goal was, quote, the most innovation and the highest level of safety. He also said that the executive order, quote, strikes a good balance between innovation and safety. At the G20, just a couple weeks ago, Michael Kratzios, the director of the White House Office of Science and Technology Policy, he called Casey, quote, our safety and security institute, end quote.

39:06 So what we're getting at here is after Mythos, after the Hugging Face incident, after the hack of OpenAI's own research cluster, safety is back, people. And it's not just back amongst the safety community who now don't have to, you know, describe themselves as security focused, et cetera. No, safety has made its way to this White House.

39:29 It is a fine word to say because when you look at AI agents autonomously hacking other companies and taking over the very infrastructure that they live on, yeah, what we're trying to do here is to advance AI safety. And I just want to highlight what I thought was such an amazing change. So the companies are now fully comfortable using this language again.

39:52 The White House is now fully comfortable using this language again. My strong hope here is that the fact that these words now have resonance and comfort, even in Republican political circles, means that we can actually do more in Congress. I'm hopeful that this is uh going to allow more action.

40:12 I'm hopeful as well. And you mentioned G20 there with Michael Kratzios. That was at the beginning of September, where the U.S. hosted leaders from around the world in Chapel Hill, North Carolina, to reaffirm its vision for AI regulation amidst these changing tones about AI. So, Greg, can you fill us in on what happened here at G20 and how that connects to this changing stance on AI regulation?

40:35 Yeah. So the G20 is an international organization mostly around convening, but convenes the 20 largest economies in the world to come together and talk policy on all kinds of issues that require international collaboration. So a ministerial meeting is when the ministers from those 20 countries get together.

40:59 And Michael Kratios, as the White House director of science and technology policy, he is routinely, and his predecessors also routinely were the ones who sit in the technology ministerial or the science and technology ministerials. So for all intents and purposes, Michael Kratios is the science technology minister of the United States of America, even though he doesn't lead a full cabinet agency.

41:24 He sits in the White House. So what we had was Kratios advocating for something at this two-day summit in North Carolina, what he called, quote, the Carolina Principles for Emerging Technology. And here we saw something interesting. He asked countries to, quote, reserve new regulation for novel considerations, end quote.

41:48 Now, what does he mean by that? I'm going to say what I think he says, and a second piece of the quote I think helps. He said, quote, policymakers do not need to approach each innovation in isolation and should not treat every emerging technology as a first of its kind policy problem. So here's what I think he's trying to say here.

42:12 AI is regulated because America has regulation, right? If you kill someone, well, it's against the law to murder. And if you say, ah, but I used AI to kill someone, well, you still broke the law, right? Or if you engage in financial fraud, insurance fraud, if you rob somebody, if you do any of these things, the laws that make it illegal before AI continue to make it illegal after AI.

42:45 And so the question when you have something like AI is to what extent are the existing laws on the books just fine because they self-update? To what extent do the existing agencies need a different enforcement strategy or a different resourcing budget given the changes in technology? So, for example, scams like online scams, I think large language models, AI generation of media, voice synthesis, those have led to a different kind of attack vector that has changed the nature of the threat, right?

43:26 So scamming people out of their money was already illegal. What needed changing was probably not the regulation, just because now you can use AI to produce a large volume of pretty high quality scam calls that like convince, for example, elderly people that their children have been kidnapped and are being ransomed.

43:48 Like that's a real threat vector. It was already illegal before we had new regulations. So what was needed was not new regulations, it was a new enforcement strategy. Like, how are we going to help the Relevant government agencies identify these scammer networks and go after them, given that they're operating at new economies of scale and they're using these AI technologies to strengthen their attack vector.

44:14 So what I think Kratios is basically saying is well, he didn't say this, but here's what I think he's getting at. He's basically saying, like, don't be the EU guys. Don't be like, oh my gosh, now there's this thing called AI, therefore we need a new AI law. He's basically saying, like, look, a lot of our existing body of regulations has kind of already drawn the right balance here.

44:37 And that has been the position of the Trump administration for a long time. But what I think is new in this G20 statement is quote, reserve new regulation for novel considerations, end quote. And there I think we get to the fact that this Trump administration has changed its posture. We're seeing that with the voluntary framework for the leading edge models.

45:02 I think what we've seen with mythos testing and the private review and advanced review that the government got that really concerned them, that led them to ban mythos as a model nationwide. And also this hugging face incident that has pointed out to this administration that there is something new. There is something novel in AI, especially in frontier AI, related to safety, related to security, that does deserve novel regulation.

45:33 And again, I think that's a change in posture from what this administration had when it took office. It was pretty much anti-all regulation for AI that wasn't already on the books, as far as I could tell. Here now we have them pro-AI safety regulation to some greater or lesser extent, you know, where they'll actually fall at the end of the story.

45:58 So go back to what I said about open AI. What do you say to an administration that has changed its mind about AI safety? What do you say to an administration that has changed its mind about AI regulation? You might be tempted to dunk on them and point out how wrong they've been all along. And that is what we call the stupid position.

46:20 The right thing to say to the Trump administration as it now embraces AI safety, as it now embraces frontier AI model regulation is it's great to have you here. Thanks for coming on board. Right? Let's work together and try and get this policy right. Because this is an administration that's going to be in power at least through January 2029.

46:42 And who knows which party will assume the presidency afterwards. There's a ton of important work to be done over the next 24 months. And I commend the White House for now coming forward and embracing some kind of new regulation for the novel threats that we face and for embracing the language of AI safety, which I think is a helpful characterization of just the reality of what we're currently facing.

47:12 Thanks for walking us through that, Greg. And I think that even though a lot of people might not pay attention to what someone says at a G20 summit, I think it's really important to understand that these Carolina principles that Michael Kratios is putting forward are the clearest explanation that we have of the administration's thinking at this point.

47:33 This is an administration that has whiplashed between strategies as AI capabilities have emerged, everything from mythos and banning it when that came out, to all of a sudden facing the threat of open Chinese models and their capabilities near the frontier. So we've had this back and forth over and over again, and we've landed here at these Carolina principles, which, as you mentioned, are finally reckoning with the safety element of AI.

48:00 The one thing, however, that the administration has not wavered on is its lawsuit with anthropic and designating it a supply chain risk.

48:10 Yes, in fact, you're right. Uh the the administration has changed its mind on some big things, but one thing they appear not to have changed their mind on is that when they designated Anthropic a supply chain risk earlier this spring, they still think that's true. But you know who doesn't think that's true is the Northern California District Court.

48:31 So let's just remind folks kind of where we are in this story. Anthropic is the provider of models to the Department of Defense or Department of War, as they style themselves now, for all classified activity, right? When you're talking about six months ago this spring, there's a lot of models out there in the large language model world.

48:57 There's Gemini, there is ChatGPT, and the various models therein, and there is, of course, Anthropic and Claude. Not all of them had gone through the lengthy process to get approved to work on classified networks. According to the reporting on this story, it took Anthropic 18 months to get its facilities security clearance processed so that they could hold and use and analyze top secret classified data.

49:26 That's why the best reporting on this topic says that they were the exclusive model provider to Maven Smart System, which I think you can say is unambiguously the most important war-fighting AI system in use on planet Earth today. So Anthropic had a monopoly because it had the clearance facilities to do this kind of work.

49:51 And it was deeply integrated with Palantir, it was deeply integrated with the Department of War and the intelligence community, and it was providing really, really impactful warfighting capabilities. But when the contract renewal came around, the Department of War said, you currently have terms of service on when your models can and cannot be used.

50:13 Those terms of service imposed limitations on what we can use the models for. We need you to sign a new contract with us that permits all lawful use. Anthropic said, no, we can't do that. We still have two restrictions that we're not comfortable with. One was lethal autonomous weapons without a human in the loop deployed operationally.

50:37 They were willing to develop such systems. They were not willing to allow the deployment without a human in the loop, at least not at this stage. So those were the two restrictions. The Department of War got furious that Anthropic would not sign that. It spilled out into President Trump getting involved, designating Anthropic a supply chain risk.

50:58 And that's when it now goes to the courts. So there's two courts where this lawsuit is playing out. One is the Northern California District Court. That's the one we're going to talk about. And then the other is a DC-based court. And so both lawsuits launched by Anthropic against the Department of War, going on simultaneously.

51:19 And now, on August 27th, we got the results from this federal lawsuit from the federal judge on the first part of the story. So in a 59-page opinion, Federal Judge Lynn from the Northern District of California granted anthropic summary judgment on First Amendment retaliation, Fifth Amendment due process, and two administrative procedure counts.

51:46 Here's a quote from the opinion: The empty invocation of national security is not a blank check to punish and retaliate against government critics. Not everybody is like a legal wonk, and maybe not everybody knows what the term summary judgment is, but what that means from the perspective of the judge, at least, is that this case is not hard.

52:09 Like if you're doing a summary judgment, it means that one side is right and it is obvious which side is right. And in this case, that is the judge siding with Anthropic and stating that it is obviously right. The opinion of the judge goes on to state: quote, though the Department of War is undisputedly free to select the AI vendor of its choice, the evidence demonstrates that the broad measures imposed on Anthropic were illegal and baseless.

52:37 I have to keep going because there's a lot of juicy quotes in this court ruling. So, quote, the government's actions were based on a desire to make a public example out of Anthropic for its arrogance in criticizing the government, not based on any articulable basis to believe that Anthropic would actually sabotage its model.

52:56 End quote. And here's something that came out in the judgment that I thought was especially juicy and illuminating. And that is the fact that the government, in the course of this trial, admitted it was wrong on the initial grounds in which it designated anthropic a supply chain risk. Here's what the opinion says here.

53:20 And this is, by the way, this is the judge summarizing what happened in the course of the court proceedings and the change in posture of the government. So this is from the introduction in the court ruling. Quote: Defendants have now backed away from the thrust of their risk assessment, which relied on Anthropic having backdoor access to its technology once deployed in a national security system.

53:44 It is now clear that Anthropic undisputedly lacks any such access, and that, as defendants concede, Anthropic's technology is itself no riskier to the national security than any other black box artificial intelligence model. End quote. Wow, that is a juicy quote, right? That is the government admitted that they were wrong about the nature of the risk that this model could even conceivably pose.

54:16 And so what did they fall back to? Well, on Twitter on September 4th, or X, I should say, Emile Michael, the Undersecretary of War for Research and Engineering, said, quote, anthropic is still a designated supply chain risk at the Department of War and for the Defense Industrial Base. Thank you for your attention to this matter, end quote.

54:40 Well, you may remember that on February 27th, in his original tweet on the subject, the Trump administration gave the Department of War a six-month phase-out period to get Anthropic off of its networks. Has it done so? No. In fact, Anthropic was, according to reporting by the Washington Post, used as part of Maven's smart system for the early phases of the war against Iran.

55:10 Entirely plausible that it's still being used in Maven Smart System, still being used for the war in Iran. And so, on the one hand, the Department of War is trying to say that anthropic is so bad that we have to designate them a supply chain risk, the kind of thing that we normally do to companies that are like secretly owned by China or Russia and secretly controlled by them.

55:37 Actually, we're going to apply that to an American company that we, on the one hand, don't trust, but on the other hand, absolutely trust because we're using it every single day with top secret classified information. And upon what basis did we decide to trust them with that top secret classified information?

55:54 The 18-month review period where we secured and certified their facilities and the personnel who will work in them and the technology that they will use in these facilities. I mean, I get why the Department of War not only lost, but lost badly. This case is kind of embarrassing, frankly, that the administration is continuing to argue where they are.

56:18 So I guess, according to the under-secretary, Anthropic is still designated supply chain risk, but at the same time, they're still, even after the completion of that six-month deadline, still being trusted with top secret information at the Department of War. What a bizarre state of affairs.

56:46 There is also a pending case in a different kind of court in Washington, D.C. that is also involved in this lawsuit. Can you just give us a quick overview of what's still outstanding and what might be different in that case compared to this one?

57:00 Yeah. So, well, I think the first thing to say is even anthropic winning in Northern California does not force the government to continue buying their stuff. It leaves the purchasing decision alone. That is government discretion. I completely agree with that, right? According to the ruling itself, the ruling, quote, does not require the Department of War to use Anthropic's products or services and does not prevent the Department of War from transitioning to other artificial intelligence providers.

57:30 End quote. I think that's totally reasonable. If the government doesn't want to buy anthropic, they don't have to buy anthropic, and they can use their own best judgment as to whether or not that should be the case. The thing that is ridiculous here is designating anthropic a supply chain risk, which it appears that even the government's own lawyers don't agree with the position that they are currently defending rather ludicrously.

57:52 So that's what just happened in Northern California. The DC court is kind of a different animal because that court is actually authorized to review classified information and in a one-sided nature. So what that means is that the government can present classified information to the judge and not share that information with anthropic.

58:19 So anthropic will be accused of things, the judge will see evidence, and anthropic will not necessarily have the opportunity to rebut those accusations or to explain or interpret that evidence. And the DC court historically has given the government a wide latitude on interpretations relating to national security.

58:45 So the Northern California court said that this case is ludicrous. I agree that this case is ludicrous. But I just want to emphasize that winning in Northern California is a good sign that you're going to win in D.C., but it's by no means a guarantee that you're going to win in D.C. Let's remember, right, that part of the grounds that they won on in the Northern California case was that their First Amendment rights were violated.

59:14 I don't believe that was actually included in the filing in the DC court, which is looking at a much narrower piece of this issue. So if, for example, right, Anthropic has a lot of Chinese nationals who work at Anthropic, as does basically every other tech company, right? Including the ones that are going to continue working with the Department of War at top secret classified levels.

59:37 But Anthropic, you know, has these set of facilities and the suite of employees who are going to all be probably American citizens or green card holders, right? Because those are the kinds of people who can get top secret clearances. If, hypothetically, I'm completely making this up, right? If the Department of War had evidence that the Chinese nationals who work at Anthropic have at some point had contact with the Chinese Communist Party or with, you know, a Chinese intelligence official or something like that, right?

1:00:11 They could present that information to the judge and they could say, you see, you can't trust Anthropic. They're compromised in the following ways, blah, blah, blah, blah, blah, blah, blah. And even if that argument is really bad, even if the evidence is completely overblown and misinterpreted, Anthropic won't have an opportunity to rebut that.

1:00:33 So they're basically relying upon the judge to do their work for them in terms of assessing the credibility of that evidence. That's why I say that winning in Northern California is a good sign for winning in DC. Many of the same arguments will apply and be weighed in the DC case, but it's just too early to say what's going to happen.

1:00:54 I mean, what is going to happen is that Claude is still not available on the Pentagon's AI site, genai.mil. Now we've seen ChatGPT Mill is on there, Grok is on there, Google's Gemini is on there, Anthropic is not on there, at least according to reporting by Defense One. But Genai.mil is only approved for controlled unclassified information all the way up to impact level five.

1:01:22 What's happening at the higher impact levels, classified secret, classified top secret, the Department of War is still using Claude. It's a pretty silly state of affairs.

1:01:41 What do you think?

1:01:42 Well, here's the thing. I was gone. We missed last week because I was traveled around in Taiwan suffering from horrific jet lag. But even though this is one of our most jam-packed episodes ever, there was so much stuff that we left on the cutting room floor, including an enormous amount of news related to U.S.

1:02:00 China competition. So we'll have to be back with an episode covering all that ground in the not too distant future.

1:02:06 That we will. And for staying with us across all these new developments, our thanks goes out to our audience for listening. And as always to Greg, it's a good to have you back. We'll be back next week with a new episode. And as always, I'm sure we'll have plenty to talk about. Thanks, Greg. Take care of