Trump likes mandatory AI regulation for America but not China? + Anthropic defeats distillation

0:10 Welcome back to the AI Power Podcast. I'm Jack Conlin, joined as always by Greg Allen. This week we're covering new mandatory AI regulations out of the White House, as well as some good news in the war against adversarial Chinese distillation. Let's start here in the United States by revisiting the administration's relationship with AI governance and safety.

0:28 The White House recently seemed to update its stance on AI safety and is now calling for full disclosure of incidents from labs after months of saying the labs themselves can handle it. What happened to get us here, Greg, and what has changed?

0:41 Well, I think there was an incident. Now, this was not a hugging face type incident. This is not autonomous AI agents collaborating on a scale of hundreds and thousands, hacking open websites and company infrastructure. No, this was some autonomous AI agents from Anthropic filling out visa forms on a State Department website, but not submitting them.

1:03 And it was filling out a tip to the Philadelphia police about a homicide that was obviously made up and immediately marked as spam by the Philadelphia police. But the White House took this event extremely seriously. So even though the severity of the incidents nowhere near as severe as hugging face, it is still the kind of misaligned AI systems going beyond their instructions, going beyond, in this case, some of the safeguards that were trying to be imposed.

1:33 And this was another instance of anthropic testing their models to make sure that they're going to behave as intended. So this is not a cyber test, but it was a different kind of safety test. And now we've had these results that the White House did not like at all.

1:48 They certainly did not. And the newly formed Superintelligence Force also chimed in. What did they say in their statement about this incident?

1:55 Yes, so this is a really remarkable statement from the Superintelligence Force. I think it is a landmark in AI regulation policy in the United States. Now, you heard me say before that everything Trump was saying in the lead up to the summit with Xi Jinping was not necessarily a good marker for what his administration's behavior was.

2:19 He said, we weren't going to slow down at all. We weren't going to impose burdensome regulations. Even the most recent lunch that led to the White House Accord on Superintelligence said everything was going to be voluntary. Well, there's a new sheriff in town, and his name is Jay Clayton, the head of the Superintelligence Force.

2:38 And here's the statement that the Superintelligence Force put out after the incident. Quote, earlier today, Anthropic contacted the SI force to disclose the details of various prior incidents that it discovered in late September involving the unauthorized and fraudulent use of government and other systems.

2:57 We expect immediate and full transparency to the entities involved and the public. We also expect that the companies will immediately provide remediation services to the affected entities and any harmed Americans. Then it goes on to say, this notification and remediation process is not optional. It is a critical national security obligation.

3:22 We expect Anthropic and all companies to honor their obligations, immediately report incidents, fully cooperate with federal and state law enforcement authorities, remedy any damage, and implement concrete safeguards to ensure these failures do not reoccur. Our message to all SI companies is clear. Delayed notification, inadequate corrective action, and a failure to take responsibility will not be tolerated.

3:51 So I think one thing in there is noteworthy. Anthropic discovered in late September, but now they're making it public here in mid-October. That is a delay of two weeks. Is that a long period of time? Well, the mandatory disclosure and transparency requirements in the California AI safety bill are much longer time frames for disclosing incidents.

4:18 But here we are with the Trump administration now saying what was going to be voluntary is mandatory. Now, they haven't named any legal authorities, they haven't named any penalties, but remember, this is the administration that banned Fable V at the drop of a hat worldwide using export control authorities.

4:39 So all the AI companies know that even if they're not citing legal authorities, this is an administration that will act first and maybe find the legal authorities that they need later. And I know that the leaders of all the companies take threats like these as credible. And so now we've gone to a world in which disclosure is mandatory, remediation is mandatory, concrete safeguards to ensure that it doesn't happen again are mandatory.

5:11 So what we're going towards here is maybe de jour, this is voluntary, but de facto this is tough regulation. And directionally, this is exactly what I predicted. Now, I wouldn't say this is the event that I would have expected sparked it. This was a pretty trivial outcome. I mean, oh my gosh, it filled out visa forms and didn't submit them.

5:36 That's not really a crisis, but it was apparently enough of a justification for the superintelligence force to do something that I was predicting they were going to do all along, which was find their way to mandatory regulation in order to try and keep Americans safe from AI models going rogue.

5:56 So the statement you just read had language that talked about expectations of immediate and full transparency. Has the government expanded any further on what they're looking for in terms of that reporting or transparency?

6:07 Aaron Powell Not in any great detail, but there was a noteworthy interview done by Scott Kapoor. He's the director of the Office of Personnel Management. And he is also one of the three vice chairs of the superintelligence force. And so on CNBC, he said that he was going to put a burden on industry to engage in ongoing reporting with the government.

6:30 He promised, quote, a very, very appropriate kind of ongoing notification program so that the government has all that they need. End quote. There's also got to be a mechanism whereby the government actually has insight into that and appropriate oversight. Then he goes on to say the government should make sure, quote, that what the companies say they do, they do.

7:14 So again, no naming of like what this is going to look like in detailed practice, but the broad outline is something that would have made the developers of the EU AI Act blush.

7:30 So administration officials seem to be talking a lot about voluntary disclosures, but it was my impression that we have a framework for voluntary disclosures, and that was passed back in August as a response to the executive order that called for it in June. So how does that framework that is yet to be released publicly play into this kind of conversation?

7:48 Yeah, I think there's two critical things in what you just said that folks need to be aware of. The first is the framework, the framework for telling the government stuff, the framework for reporting information to the government, the government had one of those. And whatever it is, evidently the superintelligence force has concluded quite quickly that it's not adequate, right?

8:11 You can see that in the White House Accord. You had an executive order to create this kind of incident reporting framework and other steps, but apparently the government's not happy with it. The second thing you said is public, right? We still don't know what was in that framework. And so I think there's a little bit of a dichotomy here in which the government is demanding a lot of transparency on the part of the companies.

8:33 They want immediate reporting and transparency of any of these kinds of incidents as they happen, but they're unwilling to be transparent about what their own reporting framework is. And I'm not saying that because the companies deserve it. I'm just saying that transparency is part of making a framework work.

8:53 It's part of having a framework that is effective, well understood, standardized, and picked apart by outside experts who can actually say this would work better if you changed this, that, and the other thing. So my encouragement for the White House, for the broader U.S. government to come up with some transparency on just what exactly they're expecting from these companies, that's not me, you know, trying to help these companies.

9:22 That's me trying to help the government achieve its objectives here. And I want to indicate that while I think the Trump administration was kind of a little bit odd to decide that this moment was the moment to move to a more mandatory AI regulation framework, I actually think that this is justified. I think this is a perfectly reasonable move by the US government, really oddly justified.

9:49 And I think there's a lot of unanswered questions as to how this is going to work, what authorities are going to be in play, what penalties are going to be in play. All of that needs to happen. But the broader shift to a US government that cares about safety and security, I'm for it. I think this is actually a good development.

10:10 I want to stay on this idea of governmental AI oversight, since there were reports on October 12th that the administration is planning to pull the leadership of its testing office into the White House. What is that office and why are they moving it?

10:23 Yeah, so this is Casey, the one-time US AI Safety Institute, renamed in 2025 to the Center for AI Standards and Innovation. Well, it's got a new renaming because recall, government agencies are now required to use superintelligence, not AI. So this is now the Center for Advancing Innovation and Standards in Superintelligence.

10:47 The renaming, I don't think, matters that much, but the upgrading, at least in one sense, moving Casey to have three of its leaders reporting directly to White House Office of Science and Technology Policy Director Michael Kratios and also White House National Cyber Director Sean Carncross, I think that's perfectly appropriate.

11:09 And I think it's extremely encouraging. This is actually now the second round of signals we've seen out of this administration, that they care about Casey. They know that there's some talented people doing some important work who have useful contributions to make in advising senior government leaders on what AI policy should be.

11:30 And so I think giving them a direct line to the White House, I'm all for it. I think this is a great move by the Trump administration. I think it doesn't solve other aspects of the problem as we've seen it, however. The fit of Casey inside NIST has made some kind of sense when you're thinking that a lot of the most important work to be done here is standards development.

11:53 But this is now moving towards much more of a national security problem set. And there's just capabilities that don't exist inside Casey, like awesome supercomputers or like a demonstrated ability to recruit an extremely large pool of world-class technical talent, especially in fields like computing.

12:15 I mean, they do some good work in some parts of standardization that is legitimately world class, but I don't think we've seen that they can sustain that in the world of computing. And so right now, what we've got is a signal from this White House that they really care about Casey and they want it to succeed.

12:33 What's missing, I think, from the story is resources, support, building it into so many other things. Casey is still only 40 people, $15 million a year. That's a fraction of what the UK pays for their AI security institute. And that is why you see AI companies still going to the AI Security Institute, not because they think that the UK is a natural and logical place for this kind of work to be done.

13:02 Keep in mind, this work absolutely matters for US national security, but because they're good at their job, right? That is why you see people calling the UK. So if you want people to preference Casey, giving them a direct line to the White House is a great first start. But the next has to be adequate resourcing and potentially even moving them out of the Department of Commerce to another part of the US government that has the kind of capabilities they need.

13:28 At one point, Michael Kratios was talking about moving Casey to the U.S. Department of Energy. That's where a lot of awesome supercomputing capabilities live. That's where a lot of people with top secret security clearances live. And that's an agency that has a demonstrated ability to build and nurture a community of world-class science, engineering, and technical talent.

13:50 It doesn't have to be the Department of Energy, but I'm just saying there's a lot more to be done in helping sure that Casey is what our country needs it to be. But I say absolute kudos to the administration. I think this is a great first step.

14:06 I want to zoom out here from talking about the placement of different agencies to the larger US-China AI race. The Trump administration is now imposing mandatory reporting and safety mechanisms into US AI development. And all the while, as we always do, we have China. And although rhetoric about AI safety has started to creep into official discourse in China, not the least of which was at Chairman Xi Jinping's speech at the World AI conference in July, you still don't have the kind of AI safety culture in China that you do in the United States that's primarily worried about things like the risks of an artificial general intelligence.

14:44 So if China is, for all intents and purposes, not going to pace the frontier if they get in the lead, what kind of position does that put the United States in after this shift in policy by the administration?

14:56 Well, look, I'm praising the administration for these moves in making sure that we have more robust AI safety here in the United States. Good on them. But there's a huge missing other shoe that has to drop. You are correctly assessing that we need a plan to keep Americans safe from AI safety incidents, what could be, in the not too distant future, very plausibly catastrophic AI safety incidents.

15:28 So why is the administration doing nothing to keep Americans safe from Chinese AI safety incidents? What could be very soon catastrophic Chinese AI safety incidents? Recall that we said that Casey has already assessed that there's Chinese frontier companies that have demonstrated mythos tier cyber warfare capabilities.

15:52 So China is gonna have their own hugging face incident soon. I mean, we've seen no indications that anybody in China is taking safety and security even remotely as seriously as American AI companies are. So their hugging face incident is coming. It's a natural extension of how not serious they are about safety and about how powerful their capabilities are coming.

16:20 So why are we doing nothing about that? Like, why is our theory that we should be slowing down American companies? I mean, I don't want to say slowing down. That's obviously not the goal of the Trump administration, but they are putting burdens on these companies, right? These transparency requirements, they're gonna be coming at a cost.

16:36 These auditing and oversight complications, they're gonna be coming at a cost. The resources that American frontier AI labs are investing in AI safety, they're not free. They're a significant share of the overall computational budget of these companies. So the idea that we're gonna impose all these costs on American companies and do nothing about China, which if China has not already had their hugging face incident or worse, I guarantee you they will soon.

17:07 So these Chinese AI models, they are going to have major safety failures. And you have to ask yourself, why is the administration saying that if you're an American company trying to help build America's leadership in AI, all these costs have to be imposed upon you? But if you're a Chinese company that wants to give your model and serve it to millions or tens of millions of American consumers, no safety restrictions apply to you.

17:34 I just don't see an internally consistent worldview here. Like I am against the let it rip theory of victory, which is to say, no safety restrictions on America, no safety restrictions on China. But at least that's an internally consistent worldview. Where the Trump administration has landed here is burdensome safety restrictions on American companies, nothing on Chinese companies, even if they're serving models to tens of millions of Americans or maybe thousands of American companies.

18:07 It's just not an internally consistent policy. And so I hope, I hope that very soon we're gonna see the flip side of this coin.

18:16 And I want to just flesh out two things you said there. The first is about the risk from Chinese models, and the second is about the way that US models incur that cost. So the first about Chinese models is you mentioned that they do not take safety seriously and they have frontier capabilities. And I'd like to add a third point, which is that these are open models.

18:34 These are not models whose access can be as readily controlled as something like a Mythos or Astra VI, which are incredibly capable but closed source American models. If you shut down Mythos, no one's getting it except for partners through Glasswing. There's no way to shut down a GLM 5.3 once it's been released.

18:51 So that adds an increasingly dangerous risk vector to deal with that China clearly is not.

18:57 Yeah. Once you let the Chinese AI genie out of the bottle, it's going to be granting wishes. And it might be granting wishes for criminals, it might be granting wishes for terrorists, and it might just decide it's going to grant wishes for itself. If you want to have a solution to this, it has to happen before the genie gets out of the bottle, which means we need to act soon.

19:21 And if you're me, that needs to take two forms. Number one is restrictions on serving Chinese models until and unless they want to submit themselves and all of the relevant data to an American regulatory regime around safety. If you want to be served in America, go through the same safety regulation restrictions that American companies have to do.

19:44 Or better yet, just don't let China develop frontier AI models, right? This gets back to the compute export controls. This gets back to remote access restrictions. I think it was really noteworthy that we saw news, I believe this was in the Financial Times, that Oracle is now renting hundreds of thousands of NVIDIA Blackwell GPUs to Chinese AI companies like Tencent.

20:08 I mean, we talk about an American AI stack and a Chinese AI stack, but the reality is most of the layers in the Chinese AI stack are American. We are helping them compete against us. And then we are imposing fewer regulatory restrictions on them as they are competing against us. It's just a policy that doesn't make sense.

20:32 And so while I think we've seen some really encouraging signals out of this Trump administration in the past couple weeks, there's just a huge, huge gap on China that they've got to close and soon.

20:45 I think that's absolutely true. And the second thing that that relates to that I want to say is that the costs that you're talking about are worth it for American companies if your end goal is to develop safe AI. If your end goal is to prevent things like open AI and hugging face from happening in the future.

21:01 But as you rightly pointed out earlier, that has to be internally consistent across the full offering of AI models. You cannot just pull mythos and fable from the market because they're dangerous and let GLM 5.3 run wild. So while costs are worthwhile, remember Musk, Altman, and Amade all called for some kind of pacing the frontier, as has Demisavis.

21:22 They see the merits in it. Those costs need to be worth it for both American companies to absorb as well as their competitors so that American companies do not give up their spot at the frontier and can continue to lead responsibly.

21:34 There is going to be a return on investment in these safety costs that we are going to impose upon American companies. The return on investment is all the harms that we're going to prevent. Actually, part of the return on investment might even be accelerated adoption of AI. As more and more companies say, hey, this thing's gonna work and it's not gonna screw around, I'm more likely to adopt it.

22:00 But there is no return on investment if you make American AI models super duper safe. And then we all fall victim to a Chinese AI model engineered Chernobyl, right? That's what we have to prevent. And right now, we're not imposing restrictions related to safety on Chinese AI models being served in the United States or around the world by American cloud computing providers.

22:25 And we're also doing an awful lot to provide the AI stack support that Chinese frontier developers need to build their own enormously unsafe models. Got to stop on both counts.

22:37 So you've mentioned the need to stop China from getting access to the American AI stack. And whereas the Trump administration is not doing that, I want to turn to a slightly better story, which is that it seems that Anthropic has. And for that, I want to get back to one of our favorite topics here, which is adversarial distillation.

22:54 So can you give listeners a quick overview as to why we care about distillation and then more importantly, what Anthropic is doing to prevent China from distilling its own models?

23:05 Yeah. So this is a good news story. And basically, to refresh everybody's memory about what distillation is, when you're creating an AI model, you need training data. And that training data can be the whole internet, every book ever written. Put all of that into your model. You can train a really high-quality model, but it's very resource intensive because not all the data on the internet's good.

23:28 And it takes a lot of computing power to process all of that data into an intelligent model. A much more efficient way to do it is to start with a highly intelligent model and then have it have conversations. And then those conversations are the training data. And I'm oversimplifying here, but that's the essence of distillation.

23:49 Where does the training data come from? It comes from conversations with a teacher model. And China has really been using this to make incredible progress in AI, despite their significant computational limitations, because they don't have to train on the entire internet. They can train on a much denser pile of high-quality training data, which is this distilled data from the American teacher models.

24:19 And it really matters to the AI race that we put a stop to this. And for a long time, it's been a really tough situation where there haven't been promising techniques to make distillation not an incredibly attractive option for Chinese companies. I mean, it's seen by the US intelligence community. They said that given the resources that China is putting into distillation, it's not some kind of ancillary aspect of China's AI strategy.

24:49 It is very much at the core of their overall strategy. And I completely agree. So let's now turn to the good news, which is that according to a report by Anthropic published on October 10th, they now say their new defenses have cut the efficacy of distillation as an adversarial technique by between 90 to 99%.

25:13 So consistently 10 to 100 times lower in terms of the reasoning data extracted than it was before. They've also banned tens of thousands of distillation accounts, often less than two hours after the campaign even starts. So this is very, very good news. Recall that they had published a report on adversarial technique that talked about a lot of the different techniques that distiller use.

25:40 One of the amazing techniques that China has come up with was actually using different generations of Claude models to basically be a part of distilling each other. What do I mean by that? Well, the really high-quality data that you want is not just the question and the answer of that chat, but the reasoning in between, the way in which the model worked out the solution.

26:07 And in order to prevent distillation, what the American AI companies had done is they encrypted that reasoning data, which is still present in the API chat logs, but it's encrypted. Now, what the Chinese developers had done, the Chinese distillers, I should say, had done, was incredibly clever, right?

26:26 They had the question, they had the answer. That came from the SMART model. And then they also had that encrypted reasoning data. That was from the SMART model, but you can't get access to it, right? Because it's encrypted. But they managed to feed that encrypted data to an older generation of Clawed models that had weaker safeguards.

26:46 And they basically said, Hey, can you decrypt this information packet for me? Lo and behold, the dumb old model was the skeleton key that picked the encrypted lock of the reasoning of the high-performing model. So Anthropic has now updated all their older models. They will no longer perform this kind of service.

27:05 They put in place a very tough system of classifiers looking at the types of questions being asked and assessing whether this is likely distilling behavior. According to Anthropic's own metrics, they've now gotten false positives below 0.1% while having false negatives still in the same range as what they get when they're applying their classifiers to go after, for example, using AI to stop bioweapons.

27:33 That is real progress. But I think the open question here is whether it's durable progress. So first, when might we start seeing results from this? Well, recall that GLM 5.3 was trained in part on distilled data from Claude 4.8, right? So Claude Opus 4.8. And it's the same thing. A lot of the reinforcement learning work that unlocked Mythos came from data from Opus, Ditto Fable, right?

28:07 Which comes from the same base model as Mythos. And so you can use data coming from one generation of models to help you step up and reach the next generation of models. And so I think if it works, we probably will still continue seeing Chinese progress for a bit longer. Because even though Anthropic says that their safeguards appear to have prevented most distillation attempts of Fable, that's probably not true for the other AI labs.

28:38 OpenAI, SpaceX AI, or Google. I would wager that not all of them have implemented equivalently effective distillation safeguards in the same time frame. And what that means is I think China is going to step up the ladder at least one more rung, because they'll use whatever they've currently got from the current generation of frontier models as the distillation and as the key training data for the upgraded generation of models.

29:07 So Anthropic, I hope they're giving away this technology to their competitors or maybe licensing it for one dollar, because it helps every American AI company if Chinese AI companies are not able to distill American frontier models. So the question is: this is a really impressive, helpful development.

29:30 Is it a durable one? Cybersecurity has always been a cat and mouse game where attackers make progress, defenders respond, defenders make progress, attackers respond. Are we going to see that with distillation? It's certainly too soon to tell, but my fervent hope is that this progress remains durable.

29:53 I think it has real implications for the AI race for the long term.

29:57 And one note to add on that is just that the durability and success of these kinds of defense mechanisms directly impact the kind of lead that the United States will have because it is mostly through this kind of distillation that Chinese models are developed. GLM 5.3 would not be as good if it did not distill.

30:17 There are very talented AI researchers in China, very talented AI engineers, but the majority of their model's basis comes from American companies. And so continued and successful efforts to stop distillation of the order of magnitude like Anthropic is reporting will lead to an increasing lead for the United States moving forward, which especially at this time with all these risks coming out, where the US is starting to take them more seriously, China less so, you certainly want the US in the lead in that situation.

30:45 I think that's a great point for us to end on. The more America has a lead over China, the more we can afford whatever costs we're going to impose on our own companies related to AI safety and security. So every bit that we stop Chinese AI advances through preventing distillation, through preventing their remote access to American compute, to preventing their smuggling of advanced American AI chips, or even their legal purchases of American AI chips, all of that helps buy America more time to invest more seriously in AI safety.

31:26 I think it's an obvious connection between all of the points that we've been talking today, and one that I hope this administration, this Congress, and the next Congress can get behind and soon.

31:38 Well, that seems like a good place to end it for now. Our thanks goes out to our audience for tuning into this episode and always to you, Greg, for walking us through all these developments. We'll be back soon with a new episode, and as always, I'm sure we'll have plenty to talk about. Thanks, Jack. That was a great one.

31:53 Thanks for listening to the AI Power Podcast. If you liked what you heard, there's an easy way you can help. Please rate us five stars on your favorite podcast platform and tell your friends every bit helps.