China reaches Mythos-tier Cyber AI + White House AI Accord + Trump/Xi Debrief

0:10 Welcome back to the AI Power Podcast. I'm Jack Conlan, back with Greg Allen after another busy week in AI policy. Greg, how are you doing?

0:18 I'm just kind of letting the events of the past few weeks just blow right past me. It's been incredibly crowded in terms of AI policy work.

0:28 Yeah, when we set out to make this a weekly podcast, I thought that that might be a bit of a stretch, but this summer has been such a whirlwind that we could make an episode out of what happened just yesterday alone.

0:40 Aaron Powell We could we could very plausibly make this a five days a week podcast. I mean, with the exception that I don't think we could pull it off at our current staffing level, but you know, there's certainly enough worth talking about to allow that to happen.

0:55 Aaron Powell Absolutely. And today we're going to cover updates from yesterday's AI luncheon at the White House as well as the United Nations General Assembly last week. Then we'll turn to increasingly powerful models and their consequences as both OpenAI and Anthropic have generated headlines recently.

1:11 But before we get to anything this week, let's catch listeners up on last week's summit between President Trump and Xi Jinping of China. He was here for a state visit. AI got a little bit of a mention during each government's readout. We previewed this meeting last week, Greg. So how did it turn out?

1:26 And what, if anything, did the countries actually agree to?

1:30 So I think we basically predicted correctly that there was not going to be any major new developments following the meeting between Treasury Secretary Scott Bessant and Chinese Vice Premier Hu. And that was where we agreed to have a dialogue with the Chinese on AI incidents. We also said that there was going to be some kind of notification mechanism.

1:54 And those two things are the most substantive things that came out of the talks, at least with respect to AI. It's clear that Trump and Chinese Communist Party chairman Xi Jinping did talk about AI, but doesn't look like any meaningful progress was made. I mean, there was a list of items in the statements by the various governments that came out after the meeting.

2:18 And on China's eight deliverables, AI is number seven, right between counter-narcotics cooperation and pandas going to the Atlanta Zoo. So not especially treated like it's the high profile, high-priority issue, barely nudged it above pandas on the list, and nothing, at least in public, on chips, export controls, distillation, open models in either the US or the Chinese readout.

2:48 Now, one thing that we did get is a little bit of a formalization of the United States and China agreeing to hold an upcoming dialogue on, quote, risks and benefits, end quote, and establishing a channel for AI incidents. Now, in the US statement, it says this: quote, the two countries established the U.S.-China Superintelligence Dialogue to exchange views on risks and benefits related to SI.

3:15 The next exchange will occur by November 2026, end quote. And it also says, quote, the United States and China also agreed to establish a bilateral communication channel for SI incidents. Now, neither of these readouts defines what would qualify as an incident or says what information will be included in these communications or even what form they'll take, whether that's going to be a fax machine or a phone line, who knows?

3:43 But it does appear that at least for the time being, we are stuck with the term superintelligence. It's in the official White House readout. And then the White House even put out a statement saying, quote, the two leaders agreed to use the term superintelligence rather than artificial intelligence, end quote.

4:03 And tragically, this is just not true. And we know this because China's version came out saying, quote, the China-US AI dialogue to exchange views on risks and benefits related to AI, end quote. And it also talks about, quote, a bilateral communication channel for AI incidents, end quote. So in other words, the Trump statement says that China agreed to use superintelligence, and the Chinese statement makes no mention of the word superintelligence.

4:32 It's frankly embarrassing that the United States is spending so much time on this. Now, I'm going to go into more detail than this topic actually deserves, but just so folks understand, the term artificial intelligence was coined in 1956. And it really gets to the facets of intelligence, thinking, learning, reasoning, planning, all of those things being done by something not human, not biological.

5:03 That's all artificial intelligence means. And the field of study has been very successfully using this term for 60 years. It's a helpful term, even though it's a little bit of a moving target in common discourse. As a field of scientific research, the people in it know what it means and use the term correctly.

5:24 Now, superintelligence is also a term that really does mean something in this field. And the term was coined by Nick Bostrom in a 1998 paper of his and massively popularized by a 2014 book called Superintelligence Paths, Dangers, Strategies, that was massive in the effective altruism community, in the AI safety community, but just among the AI labs in general.

5:52 It was taken very, very seriously. And superintelligence in Nick Bostrom's formulation is artificial intelligence, but also vastly more powerful than human intelligence. And in most forms of the extrapolation of what it means, it's a general intelligence, which means it's flexible, works on a lot of different stuff.

6:14 The point here is superintelligence already means something in the world of AI. And the difference between how the Trump administration is now using it and how the field uses it is just a space between super and intelligence and capitalizing the I, so it becomes super intelligence SI. But I think the Trump administration and Trump himself is trying to rebrand artificial intelligence, which it's fair to say is extremely controversial here in the United States.

6:42 And he's using super because it's a positive adjective. In the same way he calls his bill the one big beautiful bill, or tariffs were dropped on Liberation Day, not tariff day? This is just fundamentally like a branding exercise that it's difficult to take, especially seriously. And it's clear that the Chinese aren't taking it seriously.

7:09 And one has to ask, is the rest of the AI community taking it seriously? Well, I guess we'll get to that when we get to that lunch.

7:17 What sort of reaction did we see at a Capitol Hill in response to the summit? And how might that factor into Xi Jinping's calculus moving forward?

7:27 Yes, and I think this is a huge thing to keep in mind. Just imagine for a moment that there was some agreement that the United States and China could reach on AI that was acceptable to both sides. I'm just making this up, but let's just say America gets rid of all export controls and China agrees to allow in American inspectors to all of its AI facilities to make sure they're safe and not being used for military purposes.

7:55 That's not going to happen. Nobody's going to propose that. Just imagine it for a second. Well, even if both sides liked that kind of agreement, the fact is that Trump's ability to make such an agreement is very much in doubt just by realities in terms of what the politics in the United States are right now.

8:17 Now, the Republican Party is obviously quite loyal to Trump and quite dependent on Trump. And part of the reason for that is that the upcoming midterm elections, even though President Trump is more unpopular, according to polling data, than he has ever been in either of his two presidential administrations, he nevertheless has a massive campaign finance war chest that all kinds of Republicans are relying upon his fundraising power and his financial power for their contested races.

8:53 So you might see Republicans who are repudiating some of the things that Trump is saying in public, that includes on AI and data centers, a very controversial political issue here in the United States. But after the midterms are over, it's very unclear just how loyal Republican senators and members of Congress are going to be on Trump.

9:15 Certainly they'll agree with him on most issues, but are they going to defer to him on every issue, as they did in his first year of his second term in office, pretty much. So that is very much in question. The second thing that's in question is who's going to be in charge? The Democrats are favored to win the House.

9:36 And just recently, Nate Silver, formerly the editor-in-chief of 538, changed the odds of the Democrats retaking the Senate from toss-up to lean Democratic, reflecting a 60% chance. Now, I am not the kind of person you want interpreting your polling data, but the basic point here is that Xi Jinping has to keep in his mind as he's negotiating with President Trump that America is not a dictatorship.

10:08 Trump is not the only person who matters when it comes to making a deal. I mean, anybody who has taken American history in high school will probably remember that President Woodrow Wilson basically proposed and did most of the design work on the League of Nations during the final negotiations to end World War I.

10:32 It was his idea, and he could not get the United States Senate to ratify the treaty to make the United States a member of the League of Nations. And so the difficulty in getting the United States to adhere to agreements that only a president makes as opposed to a president with the agreement of Congress, that's really tough.

10:54 And so I think a statement that Senate minority leader Chuck Schumer that he made while Trump was meeting with Xi really does bear reading. And this is a long quote, but I think it says an awful lot. Quote: Well, today, as Trump wines and dines President Xi at the White House, he has a choice to make.

11:16 Once again, bend the knee to the CCP, meaning Chinese Communist Party, or stand up to Xi and defend our national security and keep the United States ahead of China on AI. That's why today we are calling on Trump to stand up for the nation's security by once and for all blocking China's access to our AI chips and manufacturing equipment.

11:38 President Trump must draw a hard line. Do not weaken America's export controls. Do not give China greater access to the advanced chips and chipmaking equipment it needs to close the gap with the United States. Why? Because Trump is currently giving away our greatest advantage in the AI race by greenlighting the sale of advanced AI chips to China.

11:58 China has always been desperate to get our AI chips, that they have set up elaborate ways to smuggle them, something the Trump administration has failed to stop and has only made easier for them to acquire. As we've been warning for years, Trump cannot let the CCP get its hands on our most advanced chips and the manufacturing equipment needed to produce them.

12:18 Trump could take action today to make this happen. A bipartisan group of senators has introduced legislation to update and strengthen our export controls. They are the MATCH Act, the AI Overwatch Act, and the Chip Security Act. We have worked with Republican partners. Each of these is a bipartisan piece of legislation in both chambers to include this critical legislation in the NDAA negotiations.

12:43 The Senate could pass all three bills today and protect our national security and keep USAI tech out of our adversaries' hands. Now that's a remarkable statement because there was a lot of things that Schumer could have said about Trump's meeting with Xi Jinping, but he really emphasized these export controls and he really emphasized the bipartisan nature of these export controls.

13:09 And if you are Xi Jinping, you have to keep that in the back of your mind, that there is this really significant repository of opposition to Trump's policy. So imagine that Trump says, I'm gonna sell you all the chips you could possibly want. Does that mean that you, as the Chinese government, stop supporting Huawei's attempt to indigenize, stop supporting their attempt to stockpile, stop attempting, you know, stop supporting their efforts to reverse engineer?

13:39 No, because simply the chance that America would go back on any agreement that Trump reached is in and of itself enough to adequately scare the Chinese to keep working on these reverse engineering and indigenization efforts. So there's no apology that America's going to make.

14:27 Who was invited to this event? Why were they meeting? And was there anything significant that came out of this meeting?

14:32 Aaron Powell Well, I don't know if it's significant, but let's just start once more, and I apologize to folks out there listening with the term, because superintelligence, it is going to be something that you're going to continue hearing for this administration for the foreseeable future. Because right after the lunch, Trump signed an executive order officially replacing AI with superintelligence across the federal government.

14:56 Agencies must use superintelligence and SI instead of artificial intelligence in official correspondence, public communications, websites, reports, and policy documents. Here it is. It's here to stay. Now, this does not change artificial intelligence as already defined in federal law, which is in a non-trivial number of places.

15:21 So I think what we're dealing with here is analogous to the changing of the name of the Department of Defense to the Department of War. Basically, the congressional legislation still says Department of Defense. The appropriations budget still says Department of Defense. The true legal name of the entity is still Department of Defense.

15:44 But you have this executive order saying, always forevermore, call it the Department of War. And I think that's what we're going to be dealing with in superintelligence. Now, Trump is also hoping that corporate America gets on board with this term, which I think is pretty unlikely. You have all the people who attended this lunch.

16:04 If you go to their websites, do you see superintelligence on them? No. We are indeed the SI Power podcast now. Yeah, we're not changing the name of the podcast. Don't worry about that. But let's get on to the substance. As you asked, who was there? Well, President Trump actually posted on Truth Social the seating chart, which I do think was meant to be a flex in terms of communicating just exactly how he views the various people who are trying to influence him on artificial intelligence.

16:36 Jensen Huang of NVIDIA and Elon Musk of SpaceX, they're flanking the president. Vice President Vance is directly across from them. Also, you've got Frontier Labs folks. Greg Brockman, the president of OpenAI, was there. Dario Amode and Tom Brown of Anthropic were there. Microsoft, Amazon, Meta, Google, all sending CEOs, as was AMD, as was Micron.

17:03 I could go on, but I think that about does it in terms of the industry folks and the government. You had the president, vice president, and speaker of the House, Mike Johnson. You also had Treasury Secretary Scott Bessant, Commerce Secretary Howard Lutnick, NASA Administrator Jarek Isaacman, the Director of National Intelligence, Clayton.

17:23 And this is one that I thought was kind of interesting. The chair of the Federal Trade Commission, FTC, Andrew Ferguson. And that one I can only speculate, but I think it has to deal with the inclusion of antitrust as a part of the conversation of AI safety of late. You may remember that Dario Amode, in his public essay, specifically said that some of the stuff that he was proposing might need an exemption to antitrust law.

17:53 You also had Susie Wiles, the White House Chief of Staff, Michael Kratios, the director of the White House Office of Science and Technology policy, and David Sachs, who is formerly the AI czar but now is the co-chair of PCAS, the President's Council of Advisors on Science and Technology. There's other folks that I'm just skipping over for right now, but gives you a bit of a who's who in AI policy in the United States.

18:19 Notably, Pete Hegseth of the Department of War was not in attendance, even though the Department of War actually does have a lot of equities in the AI safety and security debate. Not sure what to make of that, but I thought it was noteworthy.

18:35 So you have all these leaders from industry and government in the room. They are convened to talk about AI. What do they actually agree to, or what do they put out as a result of this meeting?

18:47 Well, the group of folks who were in the room and the group of folks who were part of what the White House is calling the White House Accord on Superintelligence, it's not the same list, actually. So the deliverable of this meeting is this voluntary joint commitment. And it's out there on the internet, but it's actually, I would say, meaningful progress on AI safety, even though it's short of what I would advocate for.

19:18 So the premise in the O'Cord's own words is that, quote, every company is responsible for developing its own technology safely and in a way that builds trust with customers and the public. And then it asks every company training and deploying frontier models, which I thought was an interesting term of art, to implement four layers of controls and audits.

19:43 One, internal controls, quote, to monitor the capabilities and alignment of its models during training and deployment around areas like cybersecurity, biosecurity, and chemical threats, and to ensure that its models do not hack or access technical systems in unintended ways. Two, an internal team to, quote, ensure all of the controls monitoring and detection are operating as intended and that any issues are remediated.

20:09 Three, quote, partner with an independent external auditor or evaluator to carry out independent assessments of whether the controls, monitoring, and detection are operating as intended, end quote. And then finally, have an independent committee of the board of directors to oversee and receive supports from the internal teams and the external auditors and to ensure issues are being remediated.

20:35 And one more thing I think is probably worth including on regulation, quote, over time, it may make sense to codify these steps into laws or regulations. Regardless of whether this is required of companies, we believe that implementing these controls and audits is critical to ensuring a safe future for everyone.

20:55 So that's signed by President Trump. It was also signed by the CEO of Google, Sundar Pichai, Anthropic, Dario Amade, Mark Zuckerberg of Meta, Greg Brockman of OpenAI, Elon Musk of XAI, and Jensen Huang of NVIDIA. So I thought there was a couple things that were noteworthy. Number one, as I read, the specific point here is around training and deploying frontier models.

21:23 Well, Microsoft was recently bragging about a model that it had trained cyber capabilities. And cyber is one of the risks listed in this document. And yet Microsoft was not a signatory to this document, even though Satya Nadella was present. And then, separate from Microsoft training and deploying its own models, it also deploys a lot of other people's models, including Chinese models.

21:51 The same is true of Amazon and AWS as hyperscale cloud providers. So it really wants you to ask like, What do they mean when they say deploying frontier models? It appears to be the case that serving them through the cloud does not meet their definition of what is going to be required, who actually bears these kinds of obligations.

22:16 Zuckerberg, who historically has put Meta opposing most ideas around AI safety or AI regulation, he's generally been opposed, and quite publicly so, to a lot of this stuff. What's interesting is that he did sign on to this document, and he also put out a statement afterwards that was quoted in Politico quote, the idea isn't that this is the only thing we will ever do.

22:43 It is that this is a start and an accord that the whole industry could come to. In other words, Meta and Nvidia, typically some of the skeptics of the need for AI safety efforts, even they are now in a position where they agree that this is a good idea. Then on the other side, you have anthropic and open AI who are calling for more and more mandatory regulation, but at least everybody can agree to these voluntary commitments.

23:13 So as you mentioned, this is a voluntary agreement. What do you think that will mean moving forward for these companies that are training and deploying AI models under this framework?

23:24 Well, I think the reality is that Anthropic and OpenAI and XAI had all expressed, as well as Google to a certain and lesser extent, but those four companies to a greater or lesser extent had basically agreed with what Dario wrote in his public essay, which was more than this document calls for. And this document added three signatories: Meta, NVIDIA, and the president, but it dropped a lot of those more significant elements of what was already agreed to.

23:59 So the White House has used its convening power to bring two of the industry laggards to at least agree to some extent of additional safety practices. I think that's legitimate progress. I commend the White House for securing this. But I also don't want to overstate what has actually been accomplished.

24:23 I mean, when asked if this was going to be binding, Trump's response at a press conference after the event was, quote, I think it's morally binding, end quote. Quote, there's going to be a tremendous self-policing aspect. It's almost like a constitution in a way. And it really is a form of protection.

24:45 No, it's not. I mean, it's definitely not nothing. It definitely counts as progress, but it is not meaningful protection for the United States population from the plausible dangers of AI. Not only is this less than what Amo Day called for in his letter and what the other CEOs said that they supported as an idea, I think it's less than what we're going to get from the California safety regulation at the state level, which all of these companies are going to be subject to.

25:19 And that's true, including the executive order that Governor Gavin Newsom recently passed on this topic. So between California's on the books legislation, the executive order on this topic, and what I expect to be updated legislation that OpenAI is itself now openly calling for, it's unclear whether this voluntary request is actually more meaningful than the on-the-books mandatory accountability and transparency that we're going to get from California in the not too distant future.

25:55 Let's move from talking about domestic AI regulation to international AI governance and the UN. On September 22nd, President Trump addressed the UN General Assembly two days before his meeting with Chairman Xi Jinping of China. In that speech, he spent a couple lines outlining his view about how the globe should treat AI governance.

26:15 What did he say in that speech and how does it compare to his previous statements about AI regulation?

26:20 Well, AI was not the only thing he was talking about, as you said, but what he did say on AI was quite a doozy in terms of rhetoric. Listen to this. He told the General Assembly that the United States, quote, totally rejects international AI governance. Here's the full quote. The United States also totally rejects any attempt to construct a globalist scheme to control artificial intelligence being spoken of so much now.

26:45 Then he goes on to categorize AI risk as a hoax. Quote, these are the same people that are now saying that AI is going to kill us all, that robots are going to attack us, and that everything is going to be a total disaster, same group of people. End quote. To do what? I mean, to address biorisks, to address cyber risks, to address chemical weapons risks.

27:48 It's all in there. And so what we have here is a disparity between the style of the Trump administration and the substance of the Trump administration. And I think the style of the Trump administration is to express contempt for a lot of the ideas of AI safety. But you have to say that substantively, they have come a long way from January 2025 during the inauguration of the second Trump administration.

28:17 And then if you look at Michael Kratios's stated position also at the UN, here's what he talked about when it comes to AI, which he uses the term superintelligence. Quote, as the world leader in frontier superintelligence, the United States has taken actions to ensure that we are able to keep the frontier moving safely and responsibly.

28:41 We have engaged frontier labs on testing and evaluation of new model capabilities, established cybersecurity clearinghouses for coordination on vulnerability mitigation, and clarified the existing law enforcement tools available to address harmful uses of superintelligence. Internationally, the United States is committed to diplomacy and dialogue on emerging technologies, but international dialogue in this forum and in others cannot be allowed to drift towards global governance.

29:10 So I think that actually makes some sense. And you can kind of get the reconciliation of what is Trump's style versus Trump's substance. And here you have the position, which, again, I think this is real progress from where the Trump administration was in January 2025.

29:31 And I do just want to expand on the disparity between public and private stances, right? What we're hearing from the Trump administration and echoed in comments from Director Kratos is that they're taking a hands-off approach. In the extent to which they are involving themselves in regulation, it's a voluntary framework.

29:48 To the extent in which they are engaging on the global scale, it's to promote an American tech stack for sovereign AI. But we've seen from this administration more interventionist policies on the part of the models themselves, even this year. Can you walk us through a little bit of that contradiction and the context leading up to this moment here between the difference in a free market, laissez-faire approach of model regulation and what they've actually done?

30:12 I think the Trump administration has frequently said, you know, we don't need new laws. We just need to effectively enforce and implement the laws we have right now. And I think that is kind of a consistent approach taken by the Trump administration. He doesn't say Congress pass new laws so that I can impose new tariffs.

30:33 He says, I'm invoking IEPA, which gives me emergency economic powers, and I'm just bringing the tariffs on. So I think you have to look at the difference between what is the de jure policy on AI regulation and what is the de facto approach that this administration has taken. De jour, the updated policy is non-interference, or as you said, Jack, laissez-faire.

30:59 Here's what Trump said on Truth Social quote, we will only encourage superintelligence. We're going to encourage it, not rein it in, end quote. Well, that's what he says is the administration's policy, but look at what they've actually done. I mean, on June 12th, the U.S. government used export controls to ban the release of Fable V worldwide after the model had already been deployed.

31:28 That was about fears about cybersecurity capabilities. And then in June, the White House also asked OpenAI to limit GPT 5.6's release, to delay it to have kind of an evaluation. Again, this was a voluntary evaluation, but OpenAI very much treated it like it was a mandatory evaluation. They expressed that they didn't like what was happening, but that they were going to abide by it.

31:55 And of course they abided by it because the administration had just proven its willingness to ban models worldwide. And this, I think, is the contradiction.

32:05 So one of the reasons that President Trump has stated that he does not want to get in the way of American AI innovation is because he does not want to give China time to catch up. He said to reporters outside the White House on September 26th, quote, the United States of America is not going to be putting on brakes.

32:22 They want to stop our progress because we're leading China by a lot, and we're going to keep it that way. I would rather not integrate because we're leading by a lot. When you're leading, you don't open it up to each other. We're leading by at least a year, maybe a year and a half. Some people say two years.

32:37 I'm not looking to open it up. However, the Center for AI Standards and Innovation, Casey, put out a report that set a very different timeline for how much Chinese models trail American models, putting it instead to four months. Greg, can you walk us through that report and the anthropic report that coincided with it?

32:56 Yes. And I think both of these are a big deal. So to refresh people's memory, ZAI is one of the leading AI labs in China. We also have Moonshot, which releases the Kimi family of models. We have DeepSeek, which is probably the best-known AI company in China. But ZAI, with its GLM series of models, is really making waves in the open source community.

33:25 And its latest one, GLM 5.3, appears to be, according to both the US government and Anthropic, sort of a mythos-esque tier of capabilities when it comes to cyber. So let's read a quote from Anthropic's report on the topic. Like Claude Mythos Preview, GLM 5.3 has strong capabilities for autonomously building end-to-end cyber exploits.

33:53 But GLM 5.3 is unlike other Frontier models in that it has been released without meaningful safeguards to limit misuse. So when we say end-to-end, what they really mean is it can not only find the bug, it can actually turn that into a working cyber attack, and it can execute that cyber attack on its own.

34:15 So GLM 5.3 has been openly available to download for about a full month now. And we covered it when it was released in August. And really what we're getting out now is the formal evaluations by the companies. And unfortunately, the news is kind of bad. It's basically saying that this is the capabilities in Chinese cyber that folks like Greg Brockman warned about in his blog post not that long ago.

34:44 The Center for AI Standards Innovation said, quote, it lags the capability of the U.S. frontier by about four months, end quote. Anthropic said, quote, our capability findings broadly match Casey's, end quote. So there you have it. ZAI is officially, when it comes to cyber AI, has closed the gap between the United States and China to only four months.

35:12 That's after President Trump is saying that the United States leads China by a year, maybe a year and a half. Some people say two years. Well, I just don't think that's true. And it may be true that that is how America would lead China if China did not have such an effective distillation campaign. But the reality is that they do.

35:36 And so with distillation, China is capable of catching up to the frontier in only four months. And remember, Mythos was the model that led Jaws to drop inside the Trump administration and make them take AI safety and security seriously in a whole new way. And now that is a tool that is not just in China's arsenal, but has been made openly available to the entire world to go download right now.

36:06 Yeah, I want to press in on why this matters a little bit more. We often hear that China is four months behind, one month behind. I want to ask you, what does that mean in terms of capability? Why does it matter that China is to our capability level with something like a mythos level model? What did Glasswing actually lead to when Mythos was the frontier?

36:30 And why is it important that China is now at that level?

36:33 Sure. So to refresh people's memory, Project Glasswing was an anthropic initiative to basically make sure that mythos-like cyberpower got into the hands of the good guys before it got into the hands of the bad guys. So Anthropic did not release mythos to everyone who could go access it from Anthropic's website or through Claude or whatever.

37:00 They only made it available to first the United States government, and then in partnership with the United States government, Anthropic made it available to a select list of partners, right? We're talking U.S. allies, Fortune 5000 companies, the types of folks who it really matters to the U.S. and allied economies if they get hacked.

37:23 So we basically said, we're going to buy time to improve defenses before this type of cyberpower is available to bad actors who might use it for offense. And I remember hearing countries complaining they weren't a part of Project Glasswing. I mean, some diplomats even came to me and asked, what do we do to get to be a part of Project Glasswing?

37:48 This seems really, really important to our international security and seems foundational to our alliance with the United States. And more companies wanted to be on that list. And it was even controversial when the wrong kinds of companies were included on it. There was some controversy when a South Korean telecommunications firm, I believe, had some Chinese ownership and was nevertheless put as a part of Project Glasswing before they were quickly removed from the approved list.

38:16 So that is what we got with the months that we've had between when Mythos first came out in April 7th and today, now that GLM 5.3 is at near capabilities. So vetted defenders, Anthropic is allowing them to use not just Mythos Preview, but Mythos 5.1, the latest and greatest type of the capabilities. And the other thing that we've done is, it's fair to say, we've really strengthened the safeguards.

38:45 Recall what Dario Amade said in his essay, where trying to improve AI safety based on capabilities from the Chat GPT era was not super helpful. You didn't learn a lot about the nature of advanced AI. But with these most recent models, we really are learning a lot about how to make them safer. And that's another thing that we bought ourselves with the four months of keeping Mythos and Mythos tier cyber capabilities private.

39:14 And OpenAI ultimately made a similar choice to only release a version with significantly strengthened safeguards. But it came at a cost. There was also a cost of not making these capabilities openly available on the market. And we should think about those as well.

39:29 So now both the US government and Anthropic are saying that GLM 5.3 is as capable as Mythos was in the spring. Why is that dangerous? Why is it dangerous that an open model at Mythos level capabilities is available and does it help anybody?

39:47 Yeah, there's two parts to that story. One is how strong are the safeguards on GLM 5.3? And then the second part is how strong is it even possible for safeguards to be, given that the weights of the model are being released openly and thereby subject to fine-tuning as a strategy for eliminating safeguards.

40:09 So I think the first thing to say is that the safeguards on GLM 5.3, at least according to Anthropic, and I think Casey would agree here, are very weak. If you want GLM 5.3 to help you plan to kill people, it's not that hard to persuade GLM 5.3 to do that. And if you want it to execute cyber attacks, it's very willing to help you do that as long as you put in place the right sort of jailbreaks that it is very subject to.

40:38 But then if you're willing to actually download the weights of the model, you can use a technique called abliteration, not obliteration. So what is obliteration? Effectively, every AI model is a lot of neurons with weights and connections to them. And it's not especially easy to see inside the mind of AI models in a super precise way, but it's increasingly easy to look inside them in a vague way.

41:14 And so effectively what you do is you observe the model in action. You ask it, hey, please go execute this cyber attack for me. And you see which neurons light up. And you see the refusal neurons engaging. You see the nodes in the neural network that are involved in safeguards, that are involved in saying no.

41:40 And you do something like either deleting those connections or setting the weight assigned to those nodes to zero. Now, I'm oversimplifying here, but the key point for obliteration is that it allows for removing technical safeguards without retraining the model. It does require a little bit of technical sophistication, but not a ton.

42:04 And it definitely doesn't require the resources that would be involved in retraining the model. What that means is that bad actors can cheaply and effectively turn this model from one that already has poor safeguards to one that has absolutely no safeguards. So we are now in an era where we should expect every cyber criminal worth their salt is going to be using GLM 5.3 if they're not already.

42:39 That's where we are. And Anthropic put it this way: quote, the release of GLM 5.3 is a meaningful step change in the cyber capabilities available to attackers. And they go on to say, quote, given this evidence, we think it's likely both state and non-state actors will use models like GLM 5.3 to cause real world harm.

43:01 End quote. And so if you're concerned that there's going to be a world in which bad actors have cyber capabilities that were previously restricted to nation-state militaries and intelligence services, that's just the world we live in right now.

43:24 I do want to note the other side, though. I used to work in cybersecurity, and these kinds of capabilities are indeed a double-edged sword. In addition to giving cyber criminals this kind of capacity, it also gives those same capabilities to defenders to help them patch flaws and bolster their cybersecurity defenses.

43:43 It's my understanding that Anthropic also pointed out these upsides of this model. And you'll remember that when the OpenAI Hugging Face incident happened this past summer, because Hugging Face was not on the short list of companies that were allowed to access the most capable of models that you were just talking about.

44:00 Who did they turn to? But open source Chinese models to help them with their cyber investigations and defense. So there are these downsides and these dangers, but there's also upsides for defenders. Can you talk about that just a little bit?

44:12 You're 100% right. It's not exclusively awful. And I do think what this points out, what this whole story points out, is that if you are a cyber defender, whether you're a government, whether you're a corporation, whether you're an individual who is a plausible target, if anthropic and open AI are for whatever reason refusing to assist you in upgrading your cyber defenses to be prepared for the AI era, why wouldn't you turn to GLM 5.3 right now, as Hugging Face did with the predecessor GLM 5.2?

44:58 And Anthropic is acknowledging, as you said, quote, cyber defenders face attackers who will use every capable tool they can. And we believe defenders should be equipped with frontier models that are at least as good as those their adversaries are using. So what they're asking for is the ability to radically expand initiatives like Project Glasswing, which is hey, government, can we please make our more capable models available to more people?

45:29 So this gets back to the Trump administration says that they're not reigning in AI. Anthropic is talking about a way that they are being reigned in right now, which is they are being prevented from upgrading cyber defenses. And Anthropic was kind of in the doghouse for a while when they had extended Mythos preview access to that South Korean telecommunications firm, as I mentioned.

45:55 But the reality here is you used to be able to say use American models because they're better, because they're safer, because we're more trustworthy, and don't use Chinese AI models. But right now, the way in which we are restricting access to advanced cyber capabilities for anthropic and open AI is literally pushing other countries into the arms of China.

46:25 And this is true in a way that I think is far more significant than the commercial competition that we face with China, because these are issues like cyber that really matter to national security organizations. So here's the thing: I commend the US government, I commend anthropic and to a certain extent OpenAI when they got involved in initiatives like Project Glasswing.

46:52 I love that we were using privileged access to our enhanced cyber defenses as a carrot to give to our allies to remind them why it's so great to be a partner and friend of the United States. But the reality is we're now living through China's rebuttal, and it's a hell of a rebuttal. And so if we don't radically expand the aperture of stuff like Project Glasswing, I'm afraid that we're just pushing countries and companies into the arms of China, that's not a state of affairs that we can tolerate.

47:31 So Mythos comes out this spring. You've now mentioned that China has rebutted. I want to ask a more simple question. How did it do that? How did GLM 5.3 get so good that it's now only four months behind the US frontier?

47:45 So we don't have all the information that we would want in order to make an ironclad determination. So what I'm gonna do is go through some possible scenarios and then give you my best guess while acknowledging that it is only speculation as to how China got here. So what are those five possible explanations?

48:10 Well, the first is that Jeep AI is just full of incredibly talented people and a lot of resources, and based on that, they were able to create an awesome model entirely internally. The second is somehow they hacked into Anthropic and got access to the weights of the Mythos model and then were able to use that to create training data for their own model.

48:39 This is something that the US national security community takes seriously as a risk. Miles Brundage, for example, over at Ivery and previously on the OpenAI security team, has been for years complaining that Frontier AI labs are not taking cybersecurity seriously enough to protect the weights of their models.

49:00 So that's one scenario. Third scenario, somehow ZAI got access to Mythos via Project Glasswing. So think about like the worst case scenario of that South Korean telecommunication company. Somehow that, but way worse, right? Like we gave access to Japan, and somehow ZAI piggybacked off of access to Japan's access.

49:25 I'm just making that up. That's just the third possible scenario. The fourth scenario is that even though Fable V comes from the Mythos family of models but had all these added safeguards on it, somehow, some way ZAI was able to distill Fable V in a way that gave training data that was relevant to creating this incredibly capable GLM 5.3.

49:54 Recall that Kratios had specifically said that the Kimi K3 model included distilled data from Fable V. This model came out after Kimmy K3, so the timing lines up. I think that the strongest evidence that we have right now, at least in the public domain, points on some kind of variant of the fourth possibility.

50:18 So Anthropic already disclosed in its September 2026 threat report on pages 150 and 151, there's a case study on Jeep AI, and it describes how the company tried a distillation attack on Fable 5 and how it was successfully prevented by Anthropic. And then it says that it noticed that Jeep AI changed its vector of attack and went after OPIS 4.8, and that this basically succeeded.

50:53 So in the June to July timeframe of this year, ZAI is executing a successful distillation attack on Opus 4.8. The Anthropic Report also provides additional details on cyber stuff specifically. So it says that Jeep built capture the flag challenges from public vulnerability data sets, then ran a distillation attack against the top model of another leading US frontier lab to solve them.

51:24 Now, Anthropic does not allege this, but based on other information that they give, a plausible interpretation is that it's GPT 5.5 that is the victim of this attack. If I'm correct in my speculation that the model that Enthropic is talking about, this report was GPT 5.5, that would explain how ZAI was able to get such extraordinary cyber capabilities.

51:53 So the basic idea here is that ZAI executed a successful distillation attack on OPUS, which was probably helpful in them getting a lot of basic cyber and software engineering capabilities. Then they got a major plus up using OPUS to grade answers provided by another Frontier AI Labs model, very possibly OpenAI's GPT 5.5, which was available from April 23rd until 5.6 came out on July 9th.

52:27 That timeframe works. And GPT 5.5 was really good at cyber. According to the UK AI Security Institute, it passed 71.4% of its expert-level cyber tasks. That's actually better than what Mythos Preview did. So the basic point here is there was a period of time, a window of opportunity, wherein there was a mythos class cybermodel out there to distill.

52:55 And according to the Anthropic Report, ZAI went after some company's model. You sort of connect the dots here. That is a plausible story for how we've gotten here. So I think when you put all that together, it basically says that the United States has a meaningful lead over China. But as long as we're releasing models openly, they are, at least thus far, able to successfully distill and ride in our slipstream.

53:28 They're able to draft off of our own success. So what can we do about all of that? Well, I think it comes back to Senator Schumer's point. We have these three pieces of legislation that would really seek to cut China off from the compute necessary to execute these attacks. We also have the possibility of cutting off these models from cloud access.

53:51 And then finally, there's the question of just how much better can our safeguards get? And specifically, the types of safeguards that are relevant to preventing distillation attacks. I mean, it's been a terrible year in terms of us giving to China freely a lot of training data necessary to create their own awesome models.

54:16 And the question is in the same way that Dario has said that actually now is an incredibly fruitful time for AI safety and security research, the question then becomes can the models also give us useful insights into how to make distillation harder, how to make obliteration harder. And if we were able to strengthen those safeguards, then we could make our models more widely available and at the same time enjoy the benefits of leadership.

54:49 China has very successfully used distillation to force America's hand. The fact that they're stealing our capability and giving it away for free basically means there's no reason for us to not give it away. But if we give it away, if in particular we give away Mythos 5.1, for example, or if we give away GPT Astra's most advanced cyber capabilities, we have to have a theory of the case for how that's not going to immediately allow China to catch up quickly, and that they're not just going to give away what we spent a trillion dollars building, that they're not just going to give it away for free.

55:34 It's really time for sort of a brass tax assessment of what is America's strategy here to maintain the lead in AI.

55:43 Yeah, distillation has always been a concern for both US companies and the US government. But if we think about the joint advisory that the US intelligence community issued that we covered in our last episode, they call distillation in that advisory a core element of China's AI strategy that is pursued not just by the companies, but likely with the knowledge of the government.

56:06 And so when you have that kind of push, I agree that it is absolutely paramount that there's a coherent strategy for how to deal with this moving forward. Otherwise, a bunch of really good, equitable copies are going to be just undercutting not just US competitiveness and having the lead for having the lead's sake, but also undercutting US efforts on regulation, safety, user safety, all these other things that come as a consequence of having leading models, those will also be undercut.

56:33 So it's absolutely paramount that people think seriously about those kinds of issues. I want to pivot here to some of those advanced models that you were just speaking about and OpenAI in particular. On September 28th, the Wall Street Journal reported that OpenAI has pulled the plug on releasing its next model, GPT 6.1 Astra, which was supposed to launch in October because of its own safety testing.

56:58 That a lab withheld from releasing a model on internal safety evaluations alone is significant. So what happened here that led to this decision?

57:06 So it's worth pointing out here that the base model, Astra, OpenAI is absolutely gonna keep using that. But what they had was this planned release of 6.1, and they're not going to release that one. I mean, for all we know, this happens at OpenAI all the time. Like maybe GPT 5.1, 5.2, 5.3, maybe they had many brothers who were crazy unsafe and OpenAI chose not to release them.

57:38 It could be the case that this is actually a common occurrence at this company and at other companies, but this is the first time they're really telling the public about their choice to not release a model because of safety concerns. Could just be part of OpenAI's goal of being more transparent generally in the wake of the hugging face disaster.

57:58 But I do think we actually learned something from what OpenAI said. So the first thing they said is that the model was more deceptive. Quote, GPT 6.1 performed poorly on tests measuring alignment or how well the model adheres to what humans would like it to do. Specifically, GPT 6.1 showed higher levels of deception.

58:20 It wasn't always honest about telling the users of the actions it did or didn't take. End quote. So alignment, deception. You may remember that GPT-6 had a different problem. The problem was monitorability. And OpenAI said, well, GPT-6 is much, much less monitorable because it's so capable of manipulating its own chain of thought, and chain of thought monitoring has been a critical tool for monitoring, but we're less concerned about that because it's so, so, so well aligned.

58:57 Well, here with 6.1, we have the worst of all possible worlds. It's terrible for monitorability and it's terrible for alignment. And so I think the idea that we had reached some kind of breakthrough where alignment was going to keep getting better forever, that hypothesis, I think, at this stage is disproven.

59:15 If we want alignment to keep getting better, we're probably going to have to have new techniques and new research to constantly keep solving this problem. It's not a natural consequence of the direction that the labs were already going. And we still have the monitorability crisis, which, as we talked about in this podcast, is really, really bad.

59:35 Well, that feels like as good of a place as any to leave it. Our thanks goes as always to our audience for listening, and my thanks goes to Greg for walking us through all this. We'll be back next week with another episode, and I'm sure we'll have plenty to talk about. Thanks, Greg. Thank you.

59:48 Thanks for listening to this podcast. And if you enjoyed it, there's a way you can help. Please give us a five star rating on whatever your preferred podcast app is and help spread the word that this podcast is out there. Every bit helps.