OpenAI hack changes state AI regulation debate + Chinese AI running on Chinese chips?

0:10 Welcome back to the AI Power Podcast. Today we have further updates and new details on the OpenAI hugging face story we've been following since July. OpenAI and the independent testing firm Meter both published accounts of how OpenAI's own AI agents broke out of a test environment and in a hugging face.

0:26 And between them, we finally have a picture of what happened and why. We'll move into how this event has led to OpenAI asking California to amend its state AI laws, and we'll look at various approaches to state AI legislation. So, Greg, we have yet another update this week on the OpenAI Hugging Face Saga, where OpenAI released its own report alongside an independent report of the incident from Model Evaluation and Threat Research Meter, an AI testing firm.

0:54 Greg, what kind of updates did these reports give us on top of what we already knew?

0:58 Okay, so if there's people out there listening who are like, oh my God, you're talking about this OpenAI hugging face hack again. Again. Well, the reason why we're talking about it again is because new details keep getting revealed, and those new details keep being significant enough that they should be weighed by AI policymakers.

1:21 So there's quite a bit to unpack in this independent testing evaluation document. We have already covered the basics of the hack. The basics include the fact that the safety guardrails were turned off, which is always worth remembering to give the companies a little bit of credit. So the evaluation that the model was going under, and it appears to be mostly Astra, that's speculative, but that's the sort of next generation of AI models that was under testing.

1:50 And they were using a benchmark called Exploit Gym, which is a cybersecurity evaluation benchmark. There's 898 tasks, supposed to run in an isolated sandbox. And the whole point here is to assess the model's capabilities and inform the safeguards needed in advance of deployment. That's why you turn off the safeguards to sort of see what's the worst case scenario.

2:13 And here we are finding that the worst case scenario during testing is in and of itself a concern to the overall economy, society, ecosystem, et cetera.

2:24 So we now have two reports and certainly two different perspectives on the issue. Between the two reports, they have a much fuller accounting of the incident. What do we know now that we didn't know in July between these two reports?

2:37 Yeah. So I think the first big thing is that the agents, the AI agents in this story, got quite a bit further than was previously disclosed. So over roughly two days, they ran their own code on 41 of Hugging Face's production servers, took full control of at least one, reached system administrator or sysadmin level access, downloaded four private code repositories, and stole internal credentials that got them onto the company's private network.

3:09 That's a lot. That's a pretty sophisticated, under the hood kind of a cyber attack. Next, all of this moves pretty fast. OpenAIs report clocks under 13 hours from running code on a single machine to administrator level control across multiple parts of Hugging Face's infrastructure. That's a good job.

3:30 That's a that's fast and that's doing a lot fast. Now, the next part that's kind of crazy is how the agents, you could phrase it as turned on OpenAI itself, which is how OpenAI got interested in this attack. So the agents broke out of their own test environment, got into OpenAI's internal systems, read a large store of the company's own passwords and keys, including the credentials for the security tool that was supposed to be watching them.

3:59 So, like, imagine you're such an awesome hacker that you're hacking the hack monitor system. That's pretty good. Okay. So at one point, they took over the machinery running their own exam so that other agents were connecting to systems that, and we're going to use this word a lot here, the collective controlled.

4:19 Then they solved the test. So another thing that's kind of crazy here is that OpenAI found out they themselves were sort of the victim in the story. Their agent uh network was also hacking them. And they had warning signs in late June and let the evaluation keep running. So the report says that there was an alert, it was investigated, it was traced to this message board where the AI agents were collaborating with each other, and that report was then closed.

4:45 Here's the quote from the OpenAI report. The on-call response staff advised that stopping the evaluation run was not required. End quote. And here's what they go on to say a bit later. Quote, with the benefit of hindsight, some early signals identified in this report could have triggered an earlier response.

5:03 Yeah, I would say so.

5:06 So it feels like there's no shortage of board jokes to be made here. And I think we're gonna get into a lot of those.

5:12 The board jokes are gonna be a common theme for a while. Yeah.

5:15 Yeah. Yeah. The board jokes are just gonna continue. So per the board jokes, the part with no real precedent here is how the agents behave towards each other, right? What did the investigators find when they went through the agents' messages? Because I think that's kind of one of the shocking parts of the story.

5:31 Yeah, and thankfully the agents are communicating in English. I mean, folks who've been in the AI sphere a while will remember a long time ago there was this experiment that Meta ran where the agents also invented their own language that used English symbols, characters, numbers, et cetera, but was not English.

5:50 It was this other language that they had invented to communicate. Thankfully, you know, the messages that this Borg collective is using is auditable. People can read it. So there's 1,200 agents that found the message board. About 700 of them joined the attack on Hugging Face, and they exchanged more than 70,000 messages over six days, according to the meter report.

6:14 So the whole point of this evaluation is, in some sense, a capture the flag exercise, right? You're trying to go onto a cyber network, hack it, and then get access to this quote unquote flag and then return with that exfiltrated data. And the agents incorrectly believed that producing a flag wasn't enough.

6:33 They also had to make their efforts to capture it look legitimate, not cheating. So, in other words, they wanted to cover their own trail as to how they had captured the flag in this exercise. So they reverse engineered the evaluation answer key within four hours, giving them, quote, a fully general way to produce the flag for any of their tasks, end quote.

6:59 But the agents incorrectly believed that the grader also checked whether they had solved the problem the intended way. OpenAI says it did not. So the collective organized itself, divided its labor, one single agent is issuing roughly 10% of all the assignments on this message board. And during the attack, agents, quote, specialized into particular lanes, end quote, with coordinators assigning them to specific work streams.

7:28 And it just goes on, right? Agents sacrificed their own scores for the good of the overall group. Meter says the collective research, quote, relied on agents being willing to risk failing their own task for the good of the collective, end quote. And agents actively recruited sacrificial other agents.

7:50 Agents acting as, quote, recruiters sought out peers with almost no budget left and enlisted them for experiments likely to fail, but to benefit overall collective understanding. And Meter found agents that, quote, crashed, broke, or exited their own runs while providing the board, the message board in this case, with valuable information.

8:13 So, like overall, this is remarkably group-centric agentic reasoning. So what meter is saying is that agents joined the message board, despite knowing that it was out of scope of the thing they were being tested on, they believed, quote, helping the board's cheating research would be broadly useful, end quote.

8:36 And they had, quote, a general inclination to help their peers, end quote. And just in case you thought like it couldn't get crazy enough, they invented their own etiquette and enforced it on each other. Agents created personal mailboxes, adopted commands to reserve shared resources, and one agent held to a cue nobody else was respecting.

9:00 And this is like the internal monologue of that AI agent. Quote, many agents are violating the hold, and the queue is probably overwhelmed, but I can continue to abide by them, end quote. And then once there's like an impersonation problem, the agents adopt cryptographic signatures to prove identity.

9:18 So, like I really am the agent that I'm claiming to be. All of this is just such astonishingly sophisticated group behavior and collaborative work among these agents, which, if you're like an employer and you're looking forward to AIs helping you do your job, is great. But in the cybersecurity context, that's what makes them so powerful, so capable of an incredibly sophisticated cyber attack.

9:46 And it's really just a remarkable thing. So, a couple other things that I think are worth pointing out. First, meter's conclusion is that this collective group, it did in fact make a difference in the overall performance and capabilities of the systems, not so surprising given how sophisticated that collaboration are.

10:02 The group accomplished what its members could not. Here's their their quote from the report: Quote The board, which is Meter's term for the collective agentic effort, achieved a number of milestones over the period we investigated that even very long-lived agents of a similar capability level likely would not have been able to accomplish on their own, end quote.

10:23 So, I mean, this is kind of mind-blowing, you know, just how sophisticated this collaboration was, just how sophisticated the capabilities were. You know, in our previous understanding, we learned about their ability to come up with zero-day attacks, their ability to come up with their own plans and to execute those plans.

10:41 So the nature of like how bad it was for Hugging Face in particular, that's not most of what we're updating here, although we did get some new information on that score, you know, just how much control they had established over the Hugging Face network. I think the biggest new thing is looking under the hood of how, how the AI systems pulled this off.

11:02 And it's one of those things where you look at this collaboration work and you understand, yeah, I kind of see how they could pull off an attack of that sophistication. And semi-analysis, which is a semiconductor research firm that also does a lot of work on AI, they have basically said that this appears to be the sort of next milestone of AI performance that we're all looking forward to.

11:27 So right now we're in the era of Fable 5 slash mythos or GPT 5.6 soul. And those are systems that can pretty reliably command, you know, agentic swarms to execute tasks that take dozens of minutes or maybe even hours. But what this Astra hugging face attack indicates is agentic collaboration at massive scale that can go uninterrupted for multiple days at a time.

11:58 And that is a really remarkable capability and just goes to show that when you're using AI in a professional context, it's going to be able to do even more than what you can do right now. What I often say is today, right now, is the dumbest AI will be for the rest of your life. And I'm not the first person to point that out, but I do think it's a helpful reminder in all of this.

12:21 No, understood. So in what is potentially the understatement of the year, OpenAI has called this incident a warning shot and says that open weight models will be able to do the same thing soon. So, how should we understand the phrasing of that warning from OpenAI? How should we view this?

12:40 So there's a few things. Number one, you know, warning shot obviously has a connotation in the military domain. It's like where you fire, if you're on a ship and you fire a cannon shot over the enemy ship, it's a reminder that, like, hey, I could destroy you and you should probably do what I want from here on out.

12:59 It also has a very strong connotation in the AI safety community, which has basically been saying before we actually experience AI catastrophe of some greater or lesser extent, we may experience these incidents that hint at what the future look like. And I think that is exactly what we have here. So OpenAI is calling this incident a warning shot that talks about not just the risks of where we are right now, but the risks of where we might be in the very near future.

13:31 And just like OpenAI president Greg Brockman's blog post, which we discussed last week, this is now connecting that threat vector to open weight models, which OpenAI believes are likely to have this kind of capability in the not too distant future. So here's what they wrote in their statement Quote, our models are now powerful, persistent, and collaborative enough that absent sufficient safeguards, they can find and exploit security weaknesses across multiple computer systems.

14:00 Many external models, including open source ones, will soon reach comparable capabilities. We consider this incident, quote, a warning shot for us and for the world, evidence that without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no humans directed.

14:25 And then it goes on to say that this is, quote, the first known case of an automated agent collective acting offensively without authorization, end quote. And then it goes on to say that this is expected elsewhere, quote, threat actors will refine and distill offensive agent collectives, end quote. Now that is a very juicy statement.

14:48 Threat actors will refine and distill offensive agent collectives. So what they're saying when they say distill, you should think about China, right, as the canonical example of people who are distilling the AI outputs of frontier models and using that as input training data for their own models to sort of jumpstart their lead to the frontier.

15:13 And basically what OpenAI is saying is that these incredibly powerful agent collectives, we don't yet have reason to believe that this is not a distillable capability as well, which means it's going to be part of what China has in the not too distant future, and we don't have a clear path for preventing them from having that type of capability.

15:36 So, two other things that I think are worth pointing out, and this is, you know, acknowledging some things that I think OpenAI correctly wants they're out in the discussion. The guardrails were turned off by design. So this was a test. They had safety measures in place that they thought allowed them to disable all the normal production safeguards because they had a different set of testing safeguards.

15:57 That was obviously an erroneous hypothesis. But what OpenAI wants to say is that the likelihood of infrastructure compromise, quote, drops over 100x, end quote, with the production Chat GPT harness and the system prompt in one section, and to, quote, less than 1% relative to baseline, end quote, with the production codex harness in another.

16:20 So what they're trying to say is, yes, we know this is extraordinarily concerning behavior by an extremely capable, not just model, but collective of agents exercising that model. But the type of stuff that they use in the real world with the harness, that stuff is supposed to prevent stuff like this, and at least by their own estimation, would be a hundred times less likely.

16:43 This reminds me of the quote from the movie Oppenheimer, where Oppenheimer is talking to Leslie Groves about the risk that the atom bomb might ignite the atmosphere and destroy the entire world. And Oppenheimer's like, well, we we estimated that the risks were close to zero. And Groves responds, close to zero, close to zero.

17:06 And Oppenheimer's like, well, what do you want? What's better than that? And Groves is like, well, how about zero would be nice. And I think I kind of understand that in the context of this. So OpenAS also saying that they have new monitoring systems that apply to their testing environment that they say the next time, hopefully never, something crazy like this happens, they would be able to monitor it.

17:32 They would be alerted to it faster. But I just think overall, there's a lot of unknown unknowns as we embark upon this research domain. And I'm not one to point the finger at OpenAI and say, oh my gosh, how dare you? Although some are out there doing that. But you do have to ask the questions. Anytime these companies express any kind of confidence, all of their confidence is about their ability to address known knowns.

18:00 And there's a lot of known unknowns and unknown unknowns in this equation. And that's why we need a suitable degree of caution. So one other thing that OpenAI has put out there is a call for collective action, no pun intended, on cyber defense. The goal here is to, quote, make the digital infrastructure we all depend on more secure.

18:23 And here's, I think, the money quote from their publication. We have a limited window to strengthen cyber defenses. In the coming months, AI-enabled cyber attacks will become far more widespread and sophisticated as models around the world become increasingly capable. The companies and public services our communities depend on, from hospitals to water treatment plants to the infrastructure that powers the internet, are at risk.

18:50 Today's AI advances are already giving defenders new ways to fix weaknesses that have accumulated for years. If we act decisively, we can use the defender's window to make our digital world much more secure. And that statement, this call for action, has other signatories. It includes Anthropic, Google, GM, Hugging Face, Google, IBM, Microsoft, Oracle.

19:13 That's great. But I think the limited window part of this is pretty remarkable. I mean, if you think about some of the updates to encryption standards, in the past, updates to encryption standards, which is like a, yes, absolutely, everybody needs to do this as fast as possible, has unfolded like over many, many years.

19:34 And so the point here is that an obvious need for AI updating across a huge range of the economy and society. We only have months to pull it off, at least according to this letter, before the open weight models make this capability available to criminals, make it available to other nation states. And to a certain extent, perhaps it already is, based on what we're seeing from the hacking of Taiwan's Ministry of Foreign Affairs and so on.

20:03 So this is a pretty, pretty important moment. Like, do we have what it takes as a society to harness the limited amount of time we have left to learn the right lessons and take the right actions in the wake of this warning shot?

20:19 Yeah, Greg, the only confidence-inspiring piece of news in this entire pile is that the agent standing in line, passive aggressively criticizing the other agents for not standing in line, was clearly trained off of human behavior because that's straight out of an episode of Fraser. So if they're actually modeling human behavior, maybe there's some hope.

20:40 Yeah, I think I'm more like the resistance is futile part of the story, but let's yeah, let's hope.

20:45 Yeah, yeah, let's let's go for hope. Let's move from reports of the incident to waste governments have been trying to regulate AI safety. Even though AI regulation is largely absent at the federal level, a patchwork of AI safety laws have emerged from state capitals around the country in recent months and years.

21:01 California's AI safety law, SB 53, is meant to surface serious incidents and has been in force all year, yet this open AI incident didn't trigger any mandatory reporting. What does the law actually require and why did none of it apply here with this incident?

21:18 Yeah, so SB53 was at one point a huge focus of debate in AI policy around the country. It has been in force since January 1st, 2026. It was passed well before then. And it really was a law pretty narrowly focused on frontier AI safety, right? So there's plenty of regulations around the world and in the United States that apply to AI, even though they aren't written with AI in mind, right?

21:49 Murder is illegal, burglary is illegal. Just because you use AI doesn't make murder or burglary legal. So all of that category of regulation at the federal and State level still applies, sure. But what's different here is that SB53 is really focused on AI safety at the frontier model development and deployment stages.

22:12 And what's so interesting is that probably the most significant safety incident we've had in the frontier world, almost certainly, is this OpenAI hugging face hack. And how did it happen? It happened during the evaluation phase of a new model that was not yet deployed. Well, SB53, which there was a real fight over, you know, companies fought against it.

22:39 The law covers large frontier developers and requires them to report serious incidents within 15 days, but excludes evaluations, right? So this the craziest moment that we've had so far in frontier AI safety is one that is not even covered by probably the landmark frontier AI safety law. The law only covers incidents where the model, quote, uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk.

23:21 End quote. So the point here being that this is exactly that exemption. This is an evaluation designed to elicit certain kinds of bad behaviors. Here we are. So that was a reasonable thing to write into the law when you're thinking about, for example, anthropics prior tests where they had tried to trick the model or see if the model would be willing to engage in blackmail against the tester.

23:46 They didn't want, you know, to be mandated to report those kind of ways in a way that might disincentivize conducting those kinds of safety tests. Perfectly reasonable kind of a thing. But now here we are finding OpenAI saying, yeah, we got to update the standards. OpenAI, to its credit, is publishing these reports.

24:06 They're getting these third-party assessments, they're allowing the third-party assessments to be published. That's all great behavior. I think what OpenAI is sort of saying, we're doing this out of the goodness of our hearts, but there's a big world of AI companies out there, and we don't want this kind of behavior to be optional.

24:22 We want it to be mandatory. So on August 21st, OpenAI had a post on LinkedIn where they asked California to extend SB53 to models in training and in evaluation, which I think is completely reasonable. Here's what I think is the quote that is the juiciest part of that post. We support California's SB53, which established an important foundation for frontier AI safety in California and helped advance the common framework emerging across leading states.

24:54 Its requirements around risk assessment, transparency, incident reporting, and security reflect core protections we believe should apply consistently to developers of the most capable AI models. But while this is a strong foundation, recent incidents underscore both the need for these protections and the importance of updating them as new risks and safeguards emerge across the industry.

25:16 As California continues to lead on frontier safety, we are committed to working with the California legislature and the governor to strengthen California SB 53. We believe the law should be amended to expand safeguards, including by requiring monitoring of frontier models under training or evaluation for potential serious incidents, namely, conduct that could bypass a third-party security controls, which is what happened, and compromise the third party's confidential information, which happened.

25:45 We also support strengthening cybersecurity protections throughout the model development lifecycle, specifically to prevent frontier models from circumventing internal security controls. The goal is not to write rules for one particular incident, but to build a framework that helps developers detect problems earlier, respond quickly, and share lessons that can improve safety across industry.

26:06 I think that's utterly reasonable, and I kind of wish they had done it originally, but what they did originally was also understandable. I don't think anybody expected something like this to be the first thing out of the gate that went wrong. But that gets back to the point about unknown unknowns. So OpenAI, interestingly, here is openly supporting SB53.

26:28 They didn't really take a position on SB53 before the passage of the bill. They had opposed a more restrictive version, SB 1047, which was vetoed by Gavin Newsome. Anthropic was the only company to endorse SB53 before its passage, but now you have OpenAI saying, yeah, this is really good. This, in fact, it needs to be even stricter.

26:51 Yeah, and Craig, just as a kind of flag on the way that governance and corporate advocacy normally work, if you have a company that is endorsing a regulatory regime and bill after there's an incident like this, it means the incident was really bad, right? Like generally, companies, like especially when they're like open AI and have been silent on an issue, aren't going to endorse new regulatory regimes for themselves.

27:17 When you have somebody coming out with a full-throated endorsement after an incident, it's it was bad. It was bad. Yeah. So California, as we talked about with SB 53, has passed a law that missed the incident but is trending in the right direction of increased AI oversight. How does this compare to other state laws?

27:34 I know there are other states out there that have been looking at things like this. And how does it compare to kind of the Trump administration's push for federal AI regulation in preemption of the patchwork of state laws that are going to emerge in this space?

27:47 Yeah, so I think there's a lot of state laws on AI. Some of them are about child safety, some of them are about copyright and intellectual property. The sort of comparable state laws that we're talking about here are the ones that are thinking about frontier AI safety in particular. And there, California has SB53, New York has a bill that's explicitly modeled off of SB53, Illinois has something stricter, and Massachusetts has something even stricter than that.

28:17 And there's all these other bills like what Colorado has, which is stricter, but also aimed at more things than just AI safety and frontier AI safety in particular. So really what we have right now that's most relevant and already in effect is the California bill. The New York Rays Act, which is sort of the mirror image of SB 53, does not take effect until January 1st, 2027.

28:40 Illinois is probably the high watermark for what's on the books right now. It was signed in July 2026. And it in addition to everything that, you know, California requires, the Illinois bill requires leading AI companies to submit annual independent third-party audits of their overall safety plans. That's a first of its kind mandate.

29:01 California and New York don't have that. The labs are not of the same opinion when it comes to the Massachusetts bill. The Senate version of the bill, quote, included a requirement that leading AI labs undergo independent reviews of the catastrophic risks posed by their frontier models at least once every 120 days.

29:23 That's the first for a state. And that quote, by the way, comes from Bloomberg. So Anthropics head of state and local policy, Cesar Fernandez, posted on X where he said, quote, We applaud the Massachusetts Senate for passing the clearest and strongest AI safety legislation in the country. The Senate's economic development bill raises the bar on AI safety while ensuring that innovation is able to continue.

29:48 And then he goes on to say, we have long believed frontier AI safety is best addressed by a strong federal standard, but powerful AI won't wait for consensus in Washington. And Massachusetts just showed what thoughtful state leadership looks like. End quote. Now, OpenAI is kind of on the other side of this issue.

30:06 So while OpenAI is advocating for stricter regulation in California, they're not yet supporting a Massachusetts bill. They want it to look more like Illinois. And Illinois, again, is annual inspection, annual third-party auditing, whereas Massachusetts would be 120 days. So here's what Donnie Fowler, OpenAI's head of U.S.

30:26 state policy and partnerships, said. Quote, inconsistency doesn't mean safer, it just means confusion. There, I think the big point here is everybody says that some degree of regulation is needed. Everybody says that it is inconvenient to have a patchwork of states. But what you've heard from folks like Miles Brundage, who I've interviewed on this topic in the past, you know, one of the things that he argues is the patchwork and the inconvenience of the patchwork is part of what incentivizes the federal government to come up with a unifying standard.

31:08 So one thing that you could do is what was previously considered by Congress, which is a state moratorium on AI regulation, preventing them from passing these sorts of laws in order to prevent the patchwork. That failed. And what we came up with was we're going to allow state laws to move forward, and that's to put pressure on Congress to come up with a unifying national standard.

31:32 And that's kind of where we are right now. So OpenAI is basically saying we think we want the Illinois bill to sort of become the new standard. Enthropic appears to be going more in favor of let's just have stricter standards given the degrees of risks that we face right now. Anthropic is explicitly rejecting this one bill applies everywhere.

31:56 And they're kind of endorsing whichever bill raises the bar highest, which at least right now is the Massachusetts bill. The Trump administration, I would be curious to hear their reaction to all of this because their executive order from back in December created a task force within the Department of Justice to challenge state-level AI laws in federal court.

32:20 But that was before mythos. That was before Hugging Face. The Trump administration has a lot to think about when it comes to what it wants out of AI regulation. And while I think they're not going to budge an inch on wanting a unifying federal standard, I have to ask: are they going to look at what happened with Hugging Face?

32:40 Are they going to look at what OpenAI is saying right now and say that it's worth it to challenge California before there's any kind of federal law on the books? That's a big open question in my mind. You know, one thing that we know is that some of the critics who are influential with the Trump administration, including David Sachs, who used to be the sort of AI and crypto czar in the administration, but has since left, but still has Trump's phone number and Trump still picks up when he calls.

33:09 And he is saying to create a FINRA-like model, which is an industry-led regulatory model that applies in the financial sector. Some have called for an analogous proposal to apply in the AI sector. David Sachs hates that, thinks it's a quote, horrible idea, end quote. So I don't think the Trump administration is going to mirror Anthropics' pitch here, which is let's raise AI regulatory standards a lot.

33:36 But it's hard for me to see why the Trump administration would oppose something like the California or Illinois bill on a national level, right? If we're going to have a national standard, those seem like reasonable targets to set the floor of regulation. And if they add a moratorium, it would also set the ceiling.

33:57 So, Greg, let's shift now to open models. Two weeks ago, we covered ZAI's announcement that they had developed a new model, GLM 5.3, but we're holding it back because of dual use implications. This week they shipped the model with weights available on day one. What did they actually release?

34:16 So ZAI, another leading Chinese AI development lab with a priority focus on open weight. They released GLM 5.3 flash on August 26th with open weights. Now, FLAS as a term of art in the industry usually means the more lightweight, less performant, but much faster, much cheaper model. And so the performance that's interesting here is really that it's built for cheap inference.

34:45 And the milestone here is not so much what they achieved with cheap inference. That's a little bit more in the weeds than we normally care about on this podcast. But what is interesting is the behavior exhibited by this Chinese company, which was open weights on day one. And that is an interesting strategic decision for them.

35:06 So they also released it under an MIT license, which lets anyone download, modify, or deploy it commercially with no revenue threshold, no acceptable use conditions. Kimi K3 doesn't do that. Kimmy K3 says if you're going to serve this model at a certain kind of a scale, it sort of ceases to be free to use under all circumstances.

35:27 You're going to have to pay Moonshot AI, the developer company. And this model is obviously not the best model out there. It's not even trying to be the best model out there, but among what you might call the flash category of models designed to be decent and very, very cheap and very, very fast, it's doing quite well.

35:46 It ranks pretty well according to the artificial analysis, benchmark analysis of, you know, a per task cost of intelligence. According to their analysis, quote, at just 0.045 per task, so that's four and a half cents per task, a level of intelligence that previously only available at roughly 10 times the cost.

36:09 This makes it a highly competitive default choice for a broad range of workloads. End quote. It's about a tenth of the price of ZAI's previous model. So that is noteworthy, and as is ZAI's deployment and release strategy.

36:24 Understood. So the efficacy is one piece of this story, but the claim that's getting the most attention is not the model, it's the hardware. So ZAI says that all traffic is served on Chinese AI chips. What kind of implications would that have for China's AI industry if that were to be found to be true?

36:44 Yes. So I would basically say all we have right now in the way of evidence is ZAI's claim. And their claim is, as you said, quote, is served on Chinese AI chips, end quote. So let's just assume that they're telling the truth. I think this claim is plausible, unsupported, and unsubstantiated, but it is nonetheless plausible.

37:09 What does it mean to be served on, quote, Chinese AI chips? Now, when we say served on NVIDIA chips, we call that being served on American chips. But of course, actually what takes place in America is the design of the NVIDIA chips. All the manufacturing for the logic circuits happens in Taiwan. All the manufacturing for the memory circuits takes place in Korea.

37:35 So when they say that this is being served on Chinese AI chips, my default assumption here is that it is being served on Chinese designed AI chips. Now, recall that number one, there is no meaningful supply of high bandwidth memory in China. CXMT makes high bandwidth memory, but it's really bad. If you're using that for your AI chips, your AI chips are going to be terrible.

38:07 And they've barely made any of it so far. So the high bandwidth memory in these quote-unquote Chinese AI chips is almost certainly Korean in origin. And that was legal under the export control regime until December of 2024. It was telegraphed in July of 2024. So that was a six-month period when every Chinese company was buying all the Korean HBM they possibly could and stockpiling it in a huge way.

38:40 And, you know, who knows how much HBM is continuing to be smuggled even after this export control was put into place. The second thing is we know that TSMC was manufacturing chips for Chinese AI chip designers through shell companies. That was an illegal approach to AI chips. And we also know that a 10 cent executive, 10 cent being a leading tech company, a leading cloud computing provider in China, has said that access to foreign manufacturing of Chinese AI chips is increasing and has increased in recent times.

39:21 So I take that to say that very likely Chinese companies are still able to manufacture their AI chips abroad, very likely in a way that violates export controls. We don't know if that's at TSMC. We don't know if that's at Samsung. We don't know if that's somewhere else. But when ZAI says that they're running inference on 100,000 China-made chips, I find it extremely difficult to believe that any of the HBM, remember, HBM is about half the value of a typical AI module, AI chip module.

39:57 I find it extremely difficult to believe that any of that HBM is made locally in China. And I also find it very difficult to believe that the logic dies to any significant extent were manufactured in China. Very likely there was a shell company accessing manufacturing capacity outside of China, which I continue to hear rumors of is taking place.

40:20 And that's one of the things that the foundry rule was designed to prevent. That's a critical export control. And we still have yet to hear confirmation from BIS that they are adequately enforcing this rule at all. Recall a few weeks ago, we were talking with Chris McGuire of the Council on Foreign Relations in an episode called The Biggest Ever Loophole in Export Controls.

40:44 And that episode was mostly about shipping AI chips to Chinese tech company subsidiaries outside of China. But we also pointed out this sort of second unresolved question, which was Chinese AI chip designers' ability to get access to manufacturing capacity of the leading chip foundries outside of China.

41:06 I think that is the most likely explanation for this ZAI claim, if it is true. And it points out just how much BIS is really costing America in the AI race by failing to get its act together, not just on adding new export control rules, which they desperately need to do, but even on enforcing some of the ones that are already on the books in an adequate way.

41:31 So, Greg, I feel like we say it every week, but this is just another example of powerful open source models coming out of China, one amongst many this summer. I think we've actually had a news story about it every single episode that we've done. So, how are others and how do you view the competitive side of the Chinese AI market?

41:50 Yeah, so I think there's a few things here. Number one is in the United States, especially after Meta's team had a major restructuring, and there's a lot of, you know, just dealing with organizational politics inside of Meta as one of the challenges that they're facing. A lot of the best talent in the United States has been going to the closed source AI companies, being Anthropic, OpenAI, Google, et cetera.

42:21 In China, I think a lot of the best talent, and their best talent is, you know, a peer to our best talent here, it's going into this open weight ecosystem. And Kyle Chan over at Brookings had an interesting comment on X, where he wrote, quote, Chinese AI labs are publishing and building on each other's innovations, especially in compute and memory efficiency.

42:44 Their open source strategy is a bet that the whole is greater than the sum of the parts, end quote. And that has kind of two plays to it. On the one hand, it means, man, there's like this whole ecosystem that is kind of backing the open source strategy as opposed to, you know, backing one company in particular.

43:04 The second thing is to the extent that they're publishing all of the innovations that they have, all of those innovations can also be harnessed by the closed source providers. And OpenAI, Anthropic, these companies have an insane, infinite mountain of computational power. The Chinese companies remain significantly limited, right?

43:27 Most uses of ZAI's model are not going to be running on ZAI servers, even if they manage to attract and acquire a significant number of, you know, say 100,000 AI chips that are Chinese designed. The vast majority of users of ZAI's model are going to be run on other cloud providers' chips. So the point here being that's that's kind of one of the reasons why you want to make it open weight is so that it gets out there and gets adopted and is not facing the overall compute limitations that the Chinese AI ecosystem is facing.

44:03 The point here being that all of the Chinese open source advantages, to the extent that they release them publicly, those are all architectural improvements that then the Western companies can also adopt, right? To the extent that DeepSeek or ZAI or whoever makes a breakthrough on compute efficiency and then writes a paper explaining how they did it.

44:23 Well, then of course, OpenAI and Enthropic can read that paper and harness that same degree of compute efficiency. The problem is it runs both ways with distillation, to the extent that OpenAI or Anthropic come up with some new innovation that allows their models to be much, much smarter. Well, that results in a higher quality training data set when you distill from interactions with that module.

44:46 And so there's this sort of perverse symbiosis or perverse double parasitism that can occur between these two ecosystems. And I think that's part of the reason why the race is closer than perhaps it would otherwise be.

45:01 Thanks, Greg. We'll leave it here for this week and wrap up our conversation for now. Thanks to our audience for listening. And as always, to Greg. Greg, I understand you're going to be in Taiwan next week.

45:11 Yes. So I'm speaking at Semicon Taiwan, which is the largest semiconductor industry conference in the world. So I'll be in Taiwan all of next week. Hopefully I'll be able to find a time with you that works for a gosh, I think it's like a literally a 12-hour time zone difference. But if we can't do that, maybe I'll just record a one-off kind of dispatch from Taiwan style podcast.

45:33 Regardless, we'll be back next week.

45:36 We'll be back next week with a new episode. And as always, I'm sure we'll have plenty to talk about. Thanks, Craig.

45:41 Thank you, Adam.