[LIVE] Anthropic Distillation & How Models Cheat (SWE-Bench Dead)

Okay, we're live. We have one person. Oh, okay. Um, people will start trickling in. Thanks for coming to Sail Live number six. This is a very exciting one. I think we have a... I mean, the topics are always fun with these. It's whatever- [chuckles]... the topic of the day on our little rat-racing minds trying to keep up with AI.

But we're welcoming the latest writer that is joining the Sail Coalition, so I think this just means more content for Sail, I think. I've been a fan of Swix and a friend for a while at this point, so I'm very happy to have his content join this, and I think you've been doing great stuff recently and continuing to evolve this. So- Thank you, sir...

welcome to the team. I just think this is [chuckles] like my friends and, and colleagues in the AI media space, and it's just great to be able to support people and keep that network closer, so welcome. Yeah. Um, well, thanks for... Uh, I just wanted to say, uh, uh, thanks for joining us. It's, uh, really a pleasure to have you on here, uh, Sean, or Swix. Um, so yeah, uh, awesome.

Uh, I just, uh, coincidentally listened to your podcast about the SWIB Benchmark, um, so yeah, awesome to, you know, small world, awesome to have you here. Yeah. Tha-thanks for having me and, uh, yeah, just glad to be on and chat. Uh, I've never ever done one of these Substack live things, so I'm curious how it works, because I always think about Substack as like a newsletter platform, uh, but they wanna go multimedia.

I think the live thing, before we get to technical content, is actually good, because it gives it a different edge, and it's just like a little bit sharper when you know you're live. I think we've all done a lot of podcasts, even podcasts that are unedited, and put those later, but I think the live thing is a different element that can be tapped into nicely. So I don't know.

[chuckles] Why don't we have... Why don't we just dive into it? We're gonna start with distillation. I put, I put how models cheat in the top so we can talk about benchmarks. I think- [chuckles]... Anthropic posted this pretty spicy blog post this week. I think it was essentially detailing how they found distributed distillation, quote-unquote, "attacks" on their services from prominent Chinese labs, and I'm very unsurprised with Anthropic calling it an attack. I think that that fits with a lot of their branding. Okay, nice.

Screen share. This is what we mean. Sean, Sean Swix is such a pro. [chuckles] Um, and it's like... [chuckles] And the screen share feature was only dropped a few days ago. But essentially it's, Anthropic is detailing how they found distributed accounts across multiple Chinese labs building shaded LLMs, and described what they were doing and why Anthropic is concerned about this in their worldview of, like, AI geopolitics. And I think it's very interesting because I'm of the opinion that the Chinese labs, like, obviously should do this.

They're at a massive GPU shortage, and using APIs is way easier than generating synthetic data on their own. Mm-hmm. And I think there's a lot of- Nathan, if I may interrupt you here, maybe we should just, for the general audience, uh, just define distillation before we- Yeah... maybe dive into the details. Go, go. Yeah.

Um, yeah, so distillation, that's like a broader concept. Uh, it's not like a new concept that came up with LLMs. It's like an older concept in machine learning in general. And, uh, distillation essentially is... The, the idea is that you're taking a larger model and, uh, train it on the outputs... Sorry, you, you have a larger model, let it generate outputs and train a smaller model on these outputs of the larger model, and the idea is that you can train the smaller model more efficiently using that larger model.

And originally, I think that you just brought up the paper here. Originally, what you would do is you would train on the logits. So old-school machine learning people might remember from deep neural networks, like the logits, the outputs of the last layer that you usually work for, with them to compute the loss function across entropy term, and you would train on, on this signal.

And, um, nowadays, in the context of LLMs, it's a bit more loose, so it does not have to be these logits that you train on. It could be just the output data, synthetic data like Nathan just, um, said. So for example... It's actually a very common practice. For example, in DeepSeek, um, R1 in the paper, um, or other people do that too, other companies, they would train the flagship model, the largest model, the R1 model with, um, 671 billion parameters, and then they would, uh, create smaller variants like, uh, I forgot the numbers, but one, three billion, uh, in a smaller range, like these small, uh, models you can run locally, and they are trained on the outputs of their own larger models.

Um, I think now the thing is also of c- I mean, this is very common practice. Everyone does that when they are producing the smaller model variants. Um, now I think the, the question or the point Nathan brought up is what happens if you are a company and you generate this synthetic data from another company's LLM and then train your own model on it?

So sorry, that was just like a little interruption, but yeah, distillation in short is training a smaller model on the outputs of a larger model basically, yeah. Yeah. And I think this is even possible at the frontier. So like people distill from something like Cloud Opus to build clouds on it. Like this is gen... They're generally doing very similar things internally. They have diff- Mm-hmm...

access to different tools and, like, richer tools. And then the other context is that all of these large labs for years have had terms of service where they say that you effectively cannot use the outputs from these APIs to train something like a competitive AI model. It is vague terms. In terms of service are not a contract. Essentially in terms of service is something that can be...

You essentially are using a service, and then if a, the provider finds you violate it, they can cut off your access. That's just kind of like a basic thing. So these have not been enforced within the US much at all. I think there was one case maybe like ByteDance a year or two ago that OpenAI cut off their API.

But this was discussed so much right after ChatGPT when people were building the first open models on, like, Alpaca and things, and it's like, is OpenAI gonna come after us for doing these research models? And I It totally died down. People were worried about this for like over a year. It was kind of an inseparable discussion. So nothing really happened, and then this is like the first prominent, um, reemergence of the discussion kind of to make... I think it's because people are far more worried about AI competitiveness- Mm. Yeah... and both. But I- Can we talk- I'm curious what you guys think. Yeah.

Can we talk a second about even how they would detect, uh, because you, you said in the beginning something about, uh, a, uh, distillation attack, and you didn't say that specifically, but you kind of like implicitly put quotation marks on "attack." Um, so how would you even detect that? So I think, uh, I mean, distillation in that context, um, means really like literally just letting ChatGPT, Claude generate synthetic data, and then you collect that synthetic data and train your own model with supervised learning, uh, supervised fine-tuning on it.

But then how would you even detect that this is an distillation attack versus just an evaluation? Because, uh, right now I'm actually running, um, I mean, I'm distilling myself for chapter eight of my book, [chuckles] but I'm doing it with open weight models, so no worry, um, Anthropic, please, uh, don't worry about it.

I use the- I distill from API models for my job. [laughs] Yeah, I, I use OpenRouter right now and just distill from the, uh, DeepSeek version 3.2 model, which I think these folks are okay with that. Um, but, um, what I wanted to say is, so when I'm evaluating models, I use basically almost the same script. So when you're evaluating in a model, you have the question and you let the, uh, model generate the answer, right?

So you generate the response to your benchmark question. And, uh, in my benchmarks, I have data sets from math 500, 500 examples. I have a bigger math data set of 12,000 examples. So you're basically just running an API in a loop to let it generate these questions and, and... Sorry, the answers. But then how would a company know, okay, this person is just evaluating versus this person is now saving that data and then data training their own model? Like, you, you see what I'm saying?

Like it's the same, the same process. I think it's a scale thing. So like- Sure, sure... when you're evaluating at least the basic evals or you're gonna do it once and not do it, there's some amount where you are-- I mean, like they say stuff here, but- Mm... there's also more of it where you just are not, like you're not gonna...

I, I think most of it is quantity, and then they're gonna look at patterns across similar accounts is what they're saying. Yeah, exactly. I think- I think they're gonna see like really repetitive stuff. Yes. So I think the interesting point this leads to is like, um, I mean you can do, uh, evaluation at a large scale. If you are a big company, you wanna know whether your LLM performs very well, you have a large suite of, um, benchmarks you are gonna run.

And, uh, but then you said like maybe looking for patterns, um, so you would m- maybe one way would be, okay, um, this is a familiar question that comes up in the benchmark, so this person is maybe not stealing our answers, it's just using it for benchmark purposes. But then it means kind of like that they are looking at what you're generating there, you know, which is, uh, I mean, of course nothing is private when you are using LLMs on the internet.

Uh, the data is somewhere, you know, intermediately stored, but then it kind of like almost implies that they are checking what you use the LLM for, what you generate, which is kind of like a sensitive topic, almost like privacy-wise, right? So that's kind of like an interesting point because, I mean, of course you mentioned the terms of service that you are not allowed to distill, but you're not distilling...

So the point I'm trying to make is you're not distilling live when you are on the platform. You are doing it somewhere later. You're just letting the LLM generate answers, and I find it kind of interesting that a company would look at that, uh, even like at the scale and, you know, call you out like, "Hey, you're, you are generating too many answers here.

That's not cool," or something. You know? That's kind of a weird thing. Yeah. Uh, I, I wanted to respond a couple, uh, this is like a, a few sentences back, but, uh, actually An- uh, Anthopic has blocked US companies first before the Chinese companies. Um, it has blocked both OpenAI and xAI from, uh, uh, from, from using the models, and I think maybe explicitly accused xAI of distilling stuff. I don't, I don't know.

But definitely not like in a, in a full blog post like this. So this one is like definitely the most high profile case. Um, and, uh, yeah, and like I, I, I do think like it is actually pretty hard to distinguish from like, "Hey, I'm just running my internal benchmark, man," and of course it's gonna be very high volume [chuckles] of like a lot of the same stuff.

Uh, because, you know, especially like some benchmarks you have to run like three, four, uh, three to five times, like the, the exact same questions, right? Like- Mm... um, I, I do think like obviously if you, if you get to the hun- to the, the tens of thousands, hundreds of thousands, then you're like, you're okay, you're not just running benchmarks.

Like you, you are, um, distilling this, uh, this thing. Um- There is a good point in the chat, um, like how would the distribution of questions look like, um, if you are distilling? And I think, um, related to your point, um, at a certain point when you have a certain magnitude of answers generated, it might look sus- suspicious, but I mean, there are a lot of legit use cases.

If a company uses your, let's say, OpenAI Claude API as their own chatbot and they have a lot of customers, it's naturally a lot of answers that are generated. And so, um, they would probably look at distributions like, um, maybe, uh, you would expect a very broad distribution when you are, um, distilling because you wanna cover pretty much everything.

And when you are running benchmarks, it's maybe more specific. Uh, you're running a math benchmark, it's just math, or if you have a customer chatbot, it's more like customer answers. But yeah, I think they would maybe analyze your distribution. I feel like this is, um, kind of a weird thing to do. [laughs] I don't know. If you're a company and you're looking into your customer privacy-- like data generated, you know, like of course it's... Well, you have to expect that it's not private, but still kind of like a weird thing that they, that they do that essentially, yeah.

Um, okay, what else do we have to talk about? [laughs] I, I think is it, is it interesting? Okay, um, I did, I did, uh, okay, so one thing, uh, this is a little bit of Substack like, you know, authors back and forth. Uh, one thing I did was, uh, I threw it into NanoBanana, which is like, uh, like kind of like a decent visual, right, of- Throw it into NanoBanana 2. It's a NanoBanana 2 live pod. [laughs] Like it just- Yeah...

released 10 minute, five minutes ago. I, I had, uh, this is actually NanoBanana 2. I, I, uh, so because I'm in the early access program, they cut you over to Nano- to, to new NanoBanana, and then I couldn't access the old one. So I was like, I was trying to like do like a diff, and I couldn't do it because [laughs] I only had access to- Classic. That is classic early tester program shit. Look at the pain we have to deal with here.

[laughs] Uh, is it interesting that DeepSeek used so much less than Minimax? I think, um, Nathan, your, in your writeup you had a little bit of a comment about like, uh, I don't know- This is a political blog post in a way. Maybe not political, but they're trying to make a point that is more about making a point than the details. Like the DeepSeek thing is definitely way smaller scale.

So, okay- I think most of the labs will experiment with all the APIs they can get access to. Like data is just so important, and you're gonna have a pipeline where you could sub in any API and then run an ablation to see if it gives you performance. Yeah. An API is kind of free. Like just do it. Uh, [laughs] like- Yeah... the, the millions of exchanges is a bit more of a bet. You can, you can measure that a bit longer, and it takes a lot longer to get...

The millions of exchanges is like tens of billions or 100 billion tokens, and it takes a lot longer to actually get that out of the API, especially when they have to spread it across a ton of accounts. These accounts are all rate limited and have other problems. Like that takes longer, but this tiny one is so fast. So I don't- Hmm...

like I, I, that was generally my point that it made it clear that Anthropic's trying, trying to use the DeepSeek name as the only Chinese AI name that people in the US know. Mm. Mm. Like marketing wise, like to make it, you know, stick or to, yeah. Um, actually, you mentioned also like the, the different APIs and everything. [laughs] I'm not like sponsored by them or I have no affiliation.

I've never talked to anyone from that company, but OpenRouter, for example, is a good example where, uh, I, I've been using it a lot for the open weight models because, um, for the bigger ones, they are too big to run them locally. And what's nice is they do also offer... So it's basically just routing you through other companies' APIs, and they select automatically at that point what is the cheapest one at that point.

I sometimes get some failures, I think when it switches might sometimes it crashes, but like in my script maybe it's like, uh, something I have to fix there. But, um, so even then if you're distilling, you can do that from multiple providers. But yeah, of course, if you are wanting something from ChatGPT or Claude, it's al-always gonna go through the official one, and then it gets, I guess, suspicious. But you could also technically, um, distill a bit from open, through OpenRouter through, I mean, like through their account, your direct account.

You can make multiple accounts and, um, yeah, it's kind of interesting that they, that they track all that and then like, yeah, different topic now that you called out, um, that they call out DeepSeek, which is quite interesting. Yeah. For what it's worth, OpenRouter seems to not be using DeepSeek, uh, in most of these. These are free models. DeepSeek's not free. Yeah, yeah. I see. Yeah. Yeah. I'm, I'm using the paid API, I should also say.

I, uh, the, it's also, it's also nice they show you how much it costs and the tokens per second for different, uh, providers. So if you go to the search in the top, you can go to the different DeepSeek ones. I just like it because I do a lot of, um, model comparisons and then, uh, this one is an older model, so maybe it only has one provider.

Um, but if you go to, uh, I think DeepSeek R1 or something, or even the normal 3.2, there should be multiple providers that, uh, if you scroll down, yeah, you can see, see there are different, um, providers and, uh, different tokens per seconds, different costs. Um, so it-it's kind of like a... I, I just like that website because it's just quick to use the API and they have a, uh, OpenAI, um, like API. So [laughs] it's almost like it's not sponsored or something.

I, I just find that generally use-useful. So but, um, yeah, just a side note. Do you wanna go back to the comparison? Did you have a high level point to make there, Sway? Oh, um, okay. Uh, just a, just a couple. Um, o-one, I think, I think the timing post Moonshot releasing their stuff, post Minimax releasing their stuff, but pre DCV 4, I think that was strategic.

Um, I think that may also have factored into why Minimax was more detected, like had a higher number. Uh, so like, you know, like when you collect data is actually very important, right? And so they interrupted or they found Minimax during the training of Minimax 2.5, right? Which, I mean, we, we will confirm this later on if, if we, if we do end up doing the call with, with them.

Um, and so obviously like the number is gonna be very high 'cause they, they're like actively looking for it and then they, they banned, uh, the Minimax accounts and Minimax changed, uh, their, their things. Actually, I, I don't think that's exactly what happened. Sorry. Uh, let me correct myself. Um, while Minimax was distilling, they released Opus 4.6, uh, and they said, uh, they said that they redirected nearly half their traffic. So I'm like, this is like, okay, very, very clearly like this is them, right?

This is the same exact traffic, uh, switched to a new model the moment a m- a new model releases. Okay, cool. Uh, DeepSeek maybe wasn't doing that because they, they hadn't been, uh, working on, on their stuff actively. I don't know, right? Like it could, it could be a different thing. Or DeepSeek is just way more efficient. Like I, I get all I need from 150K. You guys- Yeah... you know, are so inefficient.

It would be so interesting if we knew the timeframe of this. Like it... Are all of these API requests within the last four weeks? Are they within the last six months? Like that's such a different nature of what is going on. Exactly, right. That's what I'm saying. Like DeepSeek was training 3.1, 3.2 like, you know, a year ago, like [laughs] Yeah. Or like, I don't know, DeepSeek OCR.

They were like, we... I guess they said what it is, but it's not that. Yeah, yeah. Mm. Yeah, yeah. Uh, but like also scale-wise, I do think, uh, yeah, Minimax is, uh, three times smaller. It's, it's just like a faster model. They don't use, uh, MLA and they don't use the DeepSeek sparse attention, but it is, I mean, just, I think it's just group query attention, but it is still a pretty snappy model.

So it's, I, I think just attractive maybe to, um, to use it. And, uh, the other one top of my head, I don't know, maybe they had like some free Free tier or something like that, um, where I think when the models come out, they sometimes offer a free usage, and that was a more recent model than, uh, I think DeepSeek, the last one was from December, the V3.2, um, yeah.

Yeah, yeah. So I, you know, may-maybe this is an irrelevant point 'cause they, they were training- Hmm... before and, uh- Hmm... before would have the, the same amount of traffic, or they're just way more efficient, right? Uh, it does bring to mind like, okay- The efficiency thing is not it. I can guarantee it. Hmm.

[chuckles] Like that is not- Yeah. I mean, there's a small chance that they... It's like there's a chance that they got the right research idea early and, like, found the right data to use, but it's not that they're, like, gonna be 10-- three X more efficient. Okay. So it's like- [chuckles]... you know, it's a timing thing, or they just actually don't use it that much. Um, uh, I mean, like, you, you play this out, like, uh, I was like, "Okay, why, why don't they share, right? Like, they're all buddies, right?

Like, what?" [chuckles] And, like, you know, it, it, it, it does come to a point where, like, okay, let's have all of China just, like, distribute it to, like, every s- [chuckles] citizen and then- I can talk about this a little bit. That we're doing- And then- There's, there's a lot of res- there's not a lot of research, but there's a few research projects trying to understand, like, how do you use distillation data.

I think SFT is the cleanest example where you're doing- Yeah... like the, you're doing this autoregressive loss on Q&A pairs. But the strongest model is not necessarily the best teacher, and most of us in this area think it's due to, like, some, um... You have to match the probabilities of the tokens to the base model. So, like, what's happening is that Qwen-dense models are the best teachers for a lot of open weight models, and I think that's because a lot of open weight models are either Qwen or have been, like, Qwen-like for a while.

So, like, Olmo learned really well from Qwen- Hmm... and obviously, like, other Qwen models did. But, like, scaling these pipelines up to use, say, GLM 4.7 or a, a bigger DeepSeek model or a more recent Big Qwen MoE, like all of these, it's a lot harder to just generate the data from the same prompts with, like, the right sampling settings and then do SFT on them and actually make the numbers go up.

Interestingly- Yeah... GPT-OSS is a pretty good teacher. But there's, like, a huge gap there where it's like, just because you have this data does not mean it's actually gonna make your model better. So you have to do the research to be like, "Oh, we learned that we get signal out of Claude. We need to get 100 billion tokens ASAP because it's- Hmm... gonna just immediately make our model better." Like- [chuckles]...

that's not a common place to be in, in modeling because this, like, weird teacher-student dynamic going on. So I could see that being different across labs. I, I can, uh- Really interesting. Yeah, I, I think also it has something to do... I mean, I noticed also if you are distilling, um, the smaller model from the same model family, it performs better.

And I think it's to your point that, um, if you have a very, very strong model, um, it might be also too different, um, or, like, if the style is too different and then it's too much of a leap for your model to adapt. Like it's too d- too different from the Q&A answers during the pre-training or- Ah. Um, and so- Yeah, you could take the distribution could be different. Yeah... you make a bigger leap.

And another thing I wanted to say about, you mentioned Olmo, and I, it's been a while since I read the paper, but you might know way better than I do. But I think you did also train on the logits, and, um, may- you correct me if I'm wrong. Um, we didn't do technical distillation. We just did- Ah. We just took the tokens. Oh, I see, I see, I see.

Okay, then that was probably a different paper. I think Google does that for their Gemma models. Yeah, they do. Um, because here there, there's also then the distinction, uh, because you mentioned Qwen and other models. Um, you can only do that for open weight models because, uh, if you do that for Claude or OpenAI, that would not work with the logits, um, because they don't provide them.

They only provide them for some tokens, like 100 or 1,000 top tokens. And so it is, in a sense, if you want to do the real, in quotation mark, distillation, it is kind of like even easier to do that from open, open weight models because you can control it. But then also, like you said, well, we need 100 billion tokens ASAP. That is not a easy thing to do because, um, even like, uh, I mean, it's like 40 tokens per second or something for these large end models when you generate answers and getting that million, uh, billions tokens, it takes time, right?

So it's almost, like, easier to start distilling from a medium model. You just g- so it's like the question, more data versus more high perfo- uh, quality data, right? So it's also like a sweet spot, um, to like an experiment itself in an ablation study, right? Yeah. Uh, I, I like that Nathan had to call it technical distillation because it is no longer the default even though it was the first. Yeah. [chuckles] Um, also- Yeah... uh, another fun fact.

Uh, as I did my Jeff Dean interview recently, uh, and I tried to get out of him, but he, he like sort of dodged it a little bit, uh, that, uh, you know, remember like G- there were actually three sizes of Gemini models. There was, uh, Nano, Pro, and Ultra, and I was like, "Where is Ultra?" Right. Uh, they keep it in a basement and they distill from it, right? Like [laughs] that's the- Interesting. Yeah, yeah.

Maybe to, uh, also is it like to [chuckles] safeguard yourself so no one can, uh, make a copy- Yeah, yeah... or, or the price also, but probably both. [chuckles] Yeah, I mean, I, I think like this is how like... I always think of like the model you deploy is never the model you train. Mm-hmm. Uh, because you, you train the dense and then you tr- then you deploy the MoE, right? Hmm. Like you, you basically always do it. Hmm.

Like at every lab. Say more. They, like do you think they're really distilling from dense models? I mean, like I, I think that, like that is like the full, like when, when you like just unlimited resources don't care about inference, just care about maxing intelligence. Why not? Yeah. I'm, I'm not 100% sure.

I think the MoEs just give you a flop. Like I don't know if that's actually how I think of gains of MoE when you have really good MoE architecture. But I do think that they have bigger models that they distill from. Yeah. And they, they train internal models different than external because the external models have been getting a lot smaller, which is- Hmm... the kind of weird thing. We don't have a good way to measure it.

Maybe, maybe Dylan will backwards figure it out in inference max, whatever the heck. They'll be able to- Yeah... model size far. But I'm always suspicious with these things also it's really like a capacity thing too, uh, how many people use the model at the same time, uh, hardware, how much is allocated and it's always, it's like a yeah, maybe a rule of thumb, but yeah, it's really tricky I think.

It's really hard to say anything from these numbers. I do think that they might start restricting models that will only be in products and not be in API. I think the whole API business is brutally competitive, and I don't have a good sense for what the defensibility of it is. I think, like, it makes sense for something like Google and Azure and-- or any existing cloud businesses to have APIs, so that's kind of a more natural transition.

But, like, the Anthropic and OpenAI API, like, the transition from their products, which are their big differentiation, whether it's ChatGPT and Cloud Code and Codex, the differ... Like, you don't get people to go use the API from that. And I think you get a lot of people that are already spending on cloud so that then go to use the APIs, which is why, like, Lambda and Nebulas, you know, are gonna have these API products.

But, like, isn't it-- if, if Claude's really worried about distillation, like, they should put the model release in Cloud Code ASAP and then just not bother with the API. [chuckles] Hmm. I don't know when that'll happen- Yeah... but it could. I do think, though, it's a big customer base, the API, uh, customer base. Any, any type of product, um, that is built on, uh, I mean, um, with LLMs, like customer chatbot types of things, but also more generally, I do think the problem, like with-- I, I don't know exactly how the plans work in Claude, but you would reach a token max where you can only get so much with your subscription.

You can, I think, buy more tokens, but I think it's just easier with the API at a certain scale. Um, and also, like, the whole, um, OpenClaude customer base, right? Because they don't allow the plan anymore in the OpenClaude context, so you have to use the API. And I do think, given how many tokens OpenClaude generates, it's actually n-not a bad business if you don't lose money on these, you know, on these tokens.

If you sell it, uh, in a, at a not subsidized price, I do think the API is actually not a bad business model. Yeah. Uh- Do you wanna take a side? Do you wanna try to tie break? I'm obviously being provocative. Well- Like, I don't really know- Yeah, yeah... but I can see it. Like, Anthropic gives Apple vibes to me. Hmm. Uh, I-I mean, like, Anthropic has a higher chance of doing this, yes. Uh, OpenAI, just because I, like, have talked to the people so much, like, I, I just don't super believe that they will have locked models, um, to, to, to products. Um, only, only out of, I guess, idealism and sort of principles rather than economic incentive. Economic incentive would agree with you that they should have, uh, private models to, to products. And, like, recently they've done this, right?

The last three GPT-5s all had Codex variants that were two to four weeks ahead, uh, inside of-- released only inside of Codex rather than- Hmm... as an API. So they're starting to get there. But, uh, just, like, constitutionally, I don't think the, the people that run these things believe in, like, locking things behind APIs because- Yeah... they have such a huge market anyway.

So they, they, like, kind of don't care, and then they also, like, they-- if, if you're genuinely, like, sort of zealot, like, if you're not trying to maximize the value of your company and genuinely just trying to spread A-AGI everywhere, uh, then, then you release the API because you just don't know what people are gonna build with it.

Yeah. I think- Hmm. Uh, one more thing, though, with the Codex, uh, thing. I-- we will have to see, I think, next time because I think this time it might also be a bit biased towards releasing it in, uh, Codex because they almost released it simultaneously with their app that they wanna promote at the moment.

So that could have been, like, um, more like they did that so that anyone checks out the app. Um, and but we'll see next time. Yeah, yeah. But there's always a two, two to four-week exclusive window. Yeah. Yeah. Um, yeah, and, uh, you know, that's, that's their right. [chuckles] Yeah, sure, sure. If you wanna promote it, wanna promote Codex, that says pretty effective. Yep.

Um, eh, we have a bunch of questions in the chat for, like, other topics. Yeah, let's- Do we wanna cover benchmarks and then this thing or- Go right ahead. What, what do you wanna do? Uh, it's, it's your sub stack. I don't know. [chuckles] Oh, no. Aw, man. It's a collective. [chuckles] It's a collective. You should just dive into what you're interested in. [chuckles] Just go for it.

Uh, we-- I mean, uh, Sebastian was interested in, like, the Sweetbench stuff. Um, so, like, you know, this past week, Sweetbench Verified died, uh, or, like, officially- Wait, wait, what do you mean by this? [chuckles] [laughs] Yeah, let's define Sweetbench first maybe. [laughs] Okay. Uh, you know, I, I happen to have the post on this. Um, so let me just remember.

So the broader topic, uh, the umbrella topic here is how do we compare which LLM is currently the best LLM. Like, uh, uh, one of the ways would be Sweetbench, basically. But then, yeah, I will maybe let you explain because you had this brilliant podcast, uh, or article. Um, I mean, is-- Okay. Where do you want me to start? You want me to, to re- just define Sweetbench, I guess?

I guess, yeah. All right. So maybe- [chuckles]... going from, uh, so basically that it is a coding benchmark. Yes. And then the Sweetbench is, like, a popular way to compare capabilities of LLMs, and then there is Sweetbench, uh, Verified. But maybe, uh, yeah, we shou-should talk a bit- Okay... more about Sweetbench first.

E- so Sweet, Sweetbench was a, was a paper out of Princeton, uh, from O-Ophir Press' group. Um, and they do a lot of good, like, code benchmarking work. Um, and so and it ha- it happened to be that he-- they just kinda drew, uh, thousands of, of example sort of open source issues and PRs that closed tho-those issues from, uh, open source.

Um, they, there's, there's a bit of selection bias here because they only focus on popular open source and only a small number of popular open source, but a large number of issues from those open source. And then they just kinda dredged up, uh, some passing tests and then some failing tests that, that you need to make pass in order to pass the score. Uh, when it, when it launched, uh, it was kind of obscure.

Uh, Devin actually was the first one to pick, choose it as a, as a benchmark to report. Uh, and then it went from, like-- Well, I think at launch it was, like, thirteen percent, and now everyone's at eighty percent, something like that. Um, Sweetbench, because it was, it was done on, like, a student budget, was very s- kind of, let's call it sloppy or whatever. Um- Um, Terminal Bench is like this now too. Like, they're, they're just aggregate.

It's, like, hard to do a g- a benchmark that is well-calibrated across topics and differently. Yeah. Yeah. It i- it is hard. It is hard. Uh, w- so, you know, uh, for the, for the small group that is watching, uh, I'm actually working on it with Cognition for, for to launch a, a new, uh, benchmark here.

Um, but yeah, so, so OpenAI was like, "Okay, guys, we're like, SWE-bench is taking off. We are gonna adopt this, but we're n- we refuse to abide by, like, the full SWE-bench. We, we're just gonna, like, actually go and, go and curate, like, 500 subsets of, of, of, uh, the or- the original SWE-bench." And they actually hired humans to, like, go and vet through.

Like, uh, I think the, uh, it, it's somewhere inside of this blog post, but basically they hired, like, three humans for every task to just vet, uh, whether the, the task was, like, high quality or not because there's a lot of slop in there. And they were like, "Okay, this is the 500 that we, uh, that we're gonna endorse and like- So it's like a curated s- subset of- Yes...

SWE-bench where 500, let's say, challenging problems that are supposedly well-defined. [chuckles] Yeah, yeah. Um, and, and, uh, what's, what's really funny is that at launch... Uh, so this was launched in 2024. At launch, uh, OpenAI could not run all of its own 500. So for a while, like, there was, like, a few s- releases from OpenAI that reported on a subset of the subset because [chuckles] they couldn't run it on their eval infrastructure.

[laughs] So, like, their numbers were higher because their, their denominator was lower, uh, which is, which is very funny. Anyway, they- Maybe, um, in that context we should say what SWE-bench kind of looks like. I think it's, like, basically like a, a code that has, uh, bugs in it, and usually the task for the LLM is to fix- Yeah... the bug in the code, right? It's, it's right here.

The whole thing's, the whole thing's open, uh, which becomes a problem in the future. But right now you can see the whole thing, right? You can see the, uh, the, the reports from the issue ID, uh, and the, the problem statements. Uh, and then you have also the, uh, the test that you're supposed to pass and fail. Um, so it's all here on Hugging Face.

Um, and you can see that it says 500. Anyway, that, I think we don't have, we don't need to get too lost in the details on, on, on sort of, uh- Mm. Yeah, just wanted to say, uh, well, like, define the context like that this is a coding benchmark, essentially 500 examples- Yes... that are available on the internet, yeah. Okay.

Uh, and then if you want more, a bit more historical context, this is, like, a step up from HumanEval, which is more on completions, right? This is, this was in my mind the first proper agentic benchmark, I guess, a- a- apart from TAO bench, where, um, they give you the, the, the problem and the, the, the end result, and they don't really specify how you're supposed to get there.

Whereas I think a lot of, like, previous, um, benchmarks like MML-Us of the world and, and HumanEvals, uh, which i- is in the coding domain also released by OpenAI, was very much, "Here's, like, the sort of the, the problem statement, and then give me the right answer immediately after without that much, uh, sort of extra files or anything that you're supposed to run."

So, um, it, it, like, the other ones were, are more autocomplete. This one is more agentic. There, that's all a spectrum obviously because you can use agents to solve autocomplete, but, uh, that's not what HumanEval [chuckles] was, was testing. Anyway, I, I just, I wanted to make sure, like, people understand that OpenAI actually put a, invested a lot of money and effort into making SWE-bench verified from- Um, related question.

How much money do you think this cost? Oh my God, don't, don't do this to me. [chuckles] Millions. I would guess order of a coup- like, it could be even a coup- few million, but probably- I, I, I'd say, yeah, I'd say a couple million. Um, I, I, you know, um, so basically you do like, uh, okay, what's the first filter pass and then like, okay, it's 500 times three because they had three, uh- Per people per-... individuals. Yeah. Yeah.

Three people per thing, and then maybe, like, a couple more sort of, uh, verification passes or whatever, right? So, like, yeah. Um, so then they were like, "Oh, uh, so this year," they were like, "oh, well, not only is it saturated because, like, progress e- e- everyone just takes turns to increment by 0.1 every time they release a new model.

It's like it's bullshit. It's obviously bullshit." Like, the, the, the in- inherent noise in just running these models varies by, like, 0.5 to, like, one- Mm... um, every time you run it. Like, [chuckles] you just choose the highest one every time. A little nitpick. I don't think it can be 0.1% because, um, like- Sure, sure... what you said before, because it's- Is this 500?... 500 examples.

I think the smallest increment is 0.2% if I- Okay. They might average- Because, like, the- Well. [chuckles] But, like, little detail. Yeah, sorry. I, I, I think in, so, uh, as we progress to the next, uh, era of benchmarking, um, the N, so this, the N here is 500, right? The N, uh, doesn't directly correlate to the percentage points because you get sub-points as well. Ah, yeah. From the rubric. Good point. Good point.

So, so it's like terminal bench, even though it has 90 something tasks, like you can, you can, you can get subdivisions, uh, less than 1%. Um, anyway, so, uh, uh, not only do they, do they have this, they, they actually audited their own. Like, they were like, "Okay, like, how come everyone is saturating at 80%?

Like, what, what's, what's up with the remaining 20%? How come everyone's, like, failing at it?" And they were like, "Oh, actually, we looked, we, we paid even more people, six people per task now with an extra team if, if any sort of, uh, positive identification is found." Um, and we were like, "59%, uh, of them cannot even be solved at all because the original benchmark was, was, like, still slop. Like s- stuff, stuff got through that was not solvable."

And I, I actually tried to, like, illustrate this in, in my post. Um, so here, this, this is an impossible test, right? Okay. So here's an example, and this is, this is the sort of value add I did on top of the, the original post. Here, here's an example of a SWE-bench verified task that passed the first round of human verification, right?

So here's the task, like, and we wanna implement Python type hints or something. We wanna see expected behavior. We wanna see a string in the output, right? So, um, if you were given this, you would, you would never pass this because the test said, "I am looking for something called get annotation, and if you don't give me this magic string get annotation, you will fail this test."

Why? So, yeah. It's way too- It's way too specific. This feels like a coding interview. Yep, yep. [chuckles] It's like, uh, yeah. Right. So, so, so this is just a bad, uh, te- task- Mm... that somehow escaped, uh, va- validation in the first- So the only way-... run of human eval... you could kind of solve it is if you're memorizing the answer. Yeah, exactly. Mm. Exactly. Which is actually a nice...

Like, I think every benchmark should include stuff like this where if, like- Like a honeypot. [chuckles] If you solve this, you're like, "Oh, shit." Like it's a canary, right? Like, it's like, oh- Mm... like, I mean, you're definitely cheating. Um- Like a sanity check. Yeah, yeah, yeah. Yeah, yeah. Anyway, so- That's actually a really nice point, yeah Yeah, so, so like I, I, I just think like it, it-- to me it's a beautiful point of like how hard it is to make evals that there was these like multiple rounds.

There was original SWE-bench, which like the, the, the Princeton kids did do initial first pass. Then there's a second pass of OpenAI doing SWE-bench verified. And then [laughs] and then like every single person that ran for the next-- SWE-bench verified for the next one point five years did not call this out. Um [laughs] until OpenAI was like, "Hey, let's, let's like look at the data."

Uh, so I think it's like really interesting. The, uh, while they were looking at this, they had a second thing, uh, that they were, they, they, they looked at the chain of thought and inside the chain of thought they found GPT 5's own chain of thought to start including, uh, information from the future, right?

Mm. Where [laughs] because it was trained on, uh... Because the problem's open source and because it was- Mm... trained on information from GitHub, um, it would, it would like use advanced knowledge of future versions of the Django version that they were using, uh, to solve the, to solve the problem.

Like they, they knew how- Definitely seen stuff like this in the real world where the models will hallucinate the new version of API even if your script isn't on it. Like I think a lot of the Hugging Face stuff is like the worst with this, where like the models just are totally goobly glopped. Like they've seen all the versions and the API has changed too much over time- Mm. Yep...

where they just fucking throw something out there. Yeah. So, so, um, the most egregious... Yeah. I mean I, I, I think like, you know, there's, there's a lot of this, right? Like n- this sort of ethical behavior. Like, okay, so you can blame things like, oh, you should not have released, um, this, the full data set in public because obviously people can train on a full data set.

But like it's not like the researchers are trying to do this. Like because these things are also open source, like it is, like any, any t-data set that touches GitHub, any training corpus that touches GitHub is gonna just eventually absorb this. And then- Yeah, yeah. And it's not even this website or the repository directly, it's a clone of this repository or someone else who has that- Yes...

develops their own open source library and has that in the unit tests or something where it's not even intentional or malicious or anything, it's like by accident you already absorbed that. Yeah. Yeah. Or, or, or a new feature that releases this edit-only feature, it gets written up in a blog post or a conference talk or something, and then it just, it just makes it in, right? Like it's, it's really funny.

Um, um, okay, so, uh, to, to me, like OpenAI could have stopped there and said, "Okay, we're done." Uh, they, they did one more extra thing, which is kind of funny. They also then ran, uh, Flash [laughs] Gemini and Opus, and this one it was more... It was like even more egregious. Okay. They just gave the task ID and just said, "Repeat the SWE-bench task to me."

And so the from task ID they can just vomit out the whole [laughs] problem statement and the solution. [laughs] These are crazy. The stuff that's in these models when you zoom in deep is really, really incredible because like these are models that are like really, really well done, but there's just so much complexity and all the pieces of the pudding that get put in the recipe.

Yes. There's just so many weird shortcuts. I also still find it fascinating that, um, I mean, of course, uh, it's like kind of by design when you're training that you memorize, uh, things because that's literally like next token prediction. But given that how big a model is, uh, on how much data it, uh, sees, and usually it sees only the data once, that it is still has enough capacity to memorize. You know?

Like it's kind of like, uh, so usually I would think, okay, I would have to train multiple epochs to be able to memorize, but no, it, it is enough maybe to include it once or twice, uh, in the training corpus, and it can do a perfect, uh, rendition or perfect orig-, uh, yeah, recap of, of what it is in, in there, which is kind of fascinating.

Even, yeah, people who don't want that, it's, you know, it's crazy. Yeah. Labs got good at this. There's essentially like a duplication level that you need at each stage of training, and it's not easy to measure. So like if you do too much at pre-training, your model forgets basic facts, and at post-training it's probably closer to these abilities, and I think that that is a thing that is not well reflected in a...

Like you could see it in evals of your knowledge tank. Yeah. This is like an art that they have probably gotten good at. Yeah, like continued pre-training does also require some revisiting of old data. Otherwise, like you said, you have the for-forgetting. But it's still fascinating to me that with such a small fraction usually, 'cause you usually use one or two, five percent for, um, uh, like pre, continued pre-training, that it's enough to have the me-me-model memorize almost everything [laughs] which is, uh, fascinating.

Yeah. I don't know. It's just like a... Still after all these years, fascinating. Yeah. I, I, I think there's... So one of the pet topics that I pursue like two, three times a year on, on my stuff is the information theory of LMS. Um, and I, I, I still think it's like super understudied, like how come you can memorize from one pass? Like- Hmm. Yeah, exactly right.

[laughs] And then, and then also like people forget like, uh, superposition, um, which is like Anthropic's, uh, original MechInturp work, also basically stuffs information inside the smaller bits that then get forgotten. But like how, how does superposition actually work? I... People-- I don't think I've seen a convincing study on, on that. Um, okay. Anyway, anyways, I, I don't know. I, I'm done on my, on my SWE bench right now.

I don't know if you have thoughts or questions or whatever, but I, I do think like, uh, this is an example of like, yeah, the model's unintentionally cheated and, and benchmarks are hard to make, uh, and we need new ones and, uh, you know, if this happens to SWE-bench verified, like, which I think is the most scrutinized benchmark in the world- I, in my recent post- What else is there?

... uh, I had like a bar plot where I showed the SWE-bench verified numbers for most models, and like you said, they were all eighty something percent, like but literally eighty point between one and nine, let's say, where there's almost zero variation. Even like something like Minimax, um, uh, two point five, which I do think is worse than, um, GPT five point two. Like no offense, it's a smaller model, it's a cheaper model. I, I don't, I...

For my usage- or based on OpenRunner, it's a little bit worse, but on this particular benchmark, it's the same. It's like th- this is like, m- I, I don't think what I'm saying is that M2 5 should get less score on SWE-bench, but I think other models should give-- get more score. But like you said, the problems are just impossible to solve. Um, but one p- uh, point I think we didn't bring up is we said that SWE-bench verified has issues, so what do we do about it?

I think there is like a SWE-bench Pro now, which is kind of- Yes... like a, I would say like verified try to fix the regular SWE-bench and Pro tries to fix, uh, verified. But I haven't looked into this. Is it like another subset or is it a completely different set of, uh, problems? Yeah, it's a new set. Uh- Right... so the, you know, SWE-bench draws from like a 22, 2022-ish, 2023-ish era of problems.

So all you do-- Y- there's a few things you do, right? One, you do priv- private public splits, right? Mm-hmm. That's super obvious. Two, you update the dates that which you draw from, and then three, you diversify the repos and the languages, right? So th- these are all just like very, very super basic f- fixes, uh, and then obviously trying to fix the testing. Um, super basic fixes to the original SWE-bench, um, which, uh, it doesn't take a genius to, to figure out, but they did the hard work and, uh- But it is in a sense also what verified meant to do. So it's not, um...

let's say people looked at this again, but it's no guarantee that it doesn't also still have issues that might be discovered later on, right? I mean, it's- No. So, so SWE-bench verified, uh, was, was a intentional subset, right? These, these guys were like, "No, no, no, we, we need to, we need to have a superset."

Uh, we-- Not even a superset, we need a total- They're different. Yeah, yeah. Yeah. But like what I was trying to say is when, uh, SWE-bench verified was developed, there were three, uh, people per task making sure the task is well defined and everything. Mm-hmm. Mm-hmm. But then two years later it turns out, no, no, this was not the case for everything. Yeah. And what I'm trying to say is it could be that SWE-bench Pro is better, but it might still have issues that- I think-...

might not be obvious right now, but maybe in one to two years once we revisit this, uh, and you see some of the failure cases, maybe we'll discover, okay, this has still some issues. So it's not a guaranteed perfect set is what I'm saying. I don't know, but it's just like a suspicion here. Uh, totally, totally.

Um, I, you know, I, I do think C, uh, Scale AI has a professional interest in making sure this, [chuckles] this data- Yeah. No, no. Yeah, but what I was trying to say is, uh, SWE-bench verified also had a professional interest to make sure- Oh-... just by accident... very different incentive. I guess they all have very different incentives.

[laughs] This one has unlimited budget. This one has basically unlimited budget because it's like literally existential to Scale AI that they have good data. [laughs] Sure. Uh, but I also, I also think it's really nice that open-- this, this team, uh, the, the evals team at OpenAI keeps endorsing, uh, Opus. Mm-hmm. [laughs] It is kind of funny. Uh, so, so yeah.

The, uh, they deprecate SWE-bench verified and then they were like, "We're gonna report SWE-bench Pro now," uh, and GPT-5 is like, you know, number one seed. Um, maybe if... Do you know if I would want to evaluate on the private data set, how would I do that? Um, do I provide the API to... Is it like an API call I have to do against Scale AI or [sighs] I don't know.

I- You get given a API key and agree to not... You have to like agree because if you don't have a, a, a agreement then you can just have to keep the data. Yeah. So the- You have to like do special hoops to make sure that you don't steal the private eval. Hmm. Yeah, my question was basically, uh, do they even let you download the data or is it more like you send the answer to them and they do the evaluation on their backend, um, so that you don't even get to download the data?

You know, like- Scale AI, okay... 'cause otherwise like you said you could... Yeah, yeah. So basically you only provide the answer, so you have your LLM generate an answer and you submit the answers and then they have like some process to evaluate on, on their thing so that their data, private data never leaves their servers, my guess.

Because otherwise someone might upload it or something like, you know, um... Yeah, I don't know. I don't, I don't have, um- Mm... I haven't tried it, so I don't, I don't, I don't really know. I'm sure you can sort of reach out, uh, to them to figure it out. Um, yeah. Anyway. I think this is good unless people have more comments that they want to add. Yeah.

I, I think this, the-- but this is only coding, right? But like there's like every other domain [chuckles] needs this. The domain coding is the hot thing right now. Yeah. But the frontier is even more expensive. I think the frontier ex- evals are even more expensive, which is like the- Yep... apex eval from Merkur. Like evals are going to cost... This is millions.

They're gonna cost tens of millions and hundreds of millions of dollars at the frontier, which is just a very strange dynamic. Whereas like there's so much about the ecosystem is forking between frontier models and then like research and other things, and trying to follow that dynamic and explain it to people, it's gonna take a lot of work. Hmm.

But yeah, coding is, I do think really interesting because that's what most people use LLMs for these days, but also it is easier to evaluate. I think once you leave like coding math it becomes a bit obscure. How do you measure the quality of the answer? You get back to, let's say, preferences I guess, which is more like a subjective thing, where coding is more objective. So it is not a bad thing to do.

I think, uh, the other day though, um, uh, Anthropic, uh, acquired another company that does like a UI type of stuff on the computer, and I think that is, uh- Minor... something where- Doesn't really matter... you know, it's just- Normal talent, normal talent flows in AI. Total nothing burger. No, I, I mean, no, that, that's... I, I'm not trying to say- I know those people [laughs]... this is like a big thing to talk about.

Uh, what I'm trying to say is like this is another interesting, um, point for evaluating LLMs on those tasks because I think a lot of people want that to... Like, like they want an LLM to control the computer and do various things, but they are harder to measure. So that will be, um, maybe two years, uh, we will have something more like benchmarks that can, that can... It's harder to specify.

It's kind of like, what is it called? Um, in programming there's unit testing and then the system testing basically like the UI testing and stuff like that. Yes. Um, was okay. Yeah, and, and so I think that is the next- Uh, maybe gonna be the next big thing Basically and, and that thing. Yeah. Yeah, yeah. Uh, GDP VAL is usually the thing that gets brought up here. Um, so if...

I'll just leave it there. I, I, I think we've, we've sort of beaten the, the- Yep... dead horse. [laughs] Yeah, the benchmarks. Yeah. [laughs] Benchmarking. But, uh, definitely GDP VAL is sorta here. Um, I'll, I'll put it that way. Okay. Yeah. Yeah, but like the big topics are essentially the distillation and the benchmarks this week, yeah. Cool. And welcoming Max to our coalition of- Yeah...

venture pals, whatever that means formally. Uh, I, I just, it just means I get to hang out with you guys, which is, uh- Yeah... what I want anyway. Um- I describe, I describe a... I mean, it's ultimately a media vehicle, and that brands and vehicles for media are actually very influential today. I think you see many companies investing in that, and I think it's important to have people that you respect and are aligned with able to amplify each other. [laughs] Yeah.

It's also nice to, um, talk to [laughs] humans because, uh, I noticed the last couple of weeks, if you go to social media, well, I think it's 50%, uh, lobsters, like Open Claw clients nowadays. I get a lot of, um, emails, but also notifications or responses that are, they look AI-generated, so it's, it's nice to also, you know, [laughs] have this human connection and actually talk, uh, to like, uh, an expert about things, yeah.

Cool. Um, there are a bunch of, like, comments. Do you, I don't know if you wanna do, like, quick hits or are, are you, like, kinda checked out in your head? I have to go to a meeting. Okay. That's why I'm trying to wrap this up. [laughs] I see, I see, I see. Okay. Well then, you know, time is yours. I know. What, what do you wanna do? Yeah. Okay. Thanks everybody. We'll see you next week. [laughs] Yeah.

Thanks everyone for joining. It was, like, a nice spontaneous, I guess, um, you know, discussion. I, I mean, it, it, it always feels, uh, nice to talk about things, and, uh, too bad we didn't get too many, uh, or we didn't get to discuss these chat, um, uh, questions because also on my screen, I probably need glasses at some point. My screen is pretty far away. I can just barely read them.

But, um, yeah, thanks everyone for commenting. Um, it, it is just nice to see also so many, um, people excited about these topics. Yeah. Peace out. Hopefully see you later. Yeah. Have a good rest of the day. Bye. Bye.

[LIVE] Anthropic Distillation & How Models Cheat (SWE-Bench Dead) | Nathan Lambert & Sebastian Raschka

Kapitelklicken springt direkt hin

Shownotes

Transkriptvom Podcast bereitgestellt · Zeitmarken springen hin