AI Did Brain Surgery on Another AI (Then $1.5T Vanished)

Christopher: It built a tool to do brain surgery.

Zach: Yeah. one point five trillion dollars disappeared from the stock market last week.

Marty Weiner: my God, the AI bubble, it's over. Everybody, let's talk about

Christopher: Yeah.

Marty Weiner: it on C N B C

Zach: That that's a daily. Hello, welcome to Last Humans in the Loop, your weekly AI news podcast. Today we are covering big news on the frontier is that models are getting really good at directly editing other models' weights, which could have implications for drastic increases in model power and even crazier, maybe could help solve alignment. in security, cursor served a malicious fake clawed code update. It was serving for 11 hours. We're also going to talk about

Marty Weiner: we should.

Zach: the tech sell-off, if we want to call it that. that's gonna include Reddit. So we're gonna have Chris, who still works for Reddit, put a gag or a tennis ball in his mouth or something.

Christopher: Shaw.

Zach: very good point. Wow, that's spicy. and then we're gonna talk about robots. Is the US government going to restrict imports of robots and the parts we need to make them in the United States? I'm worldwide celebrity Zach Wienersmith. I'm here with my favorite Reddit CTO. Chris Slow, and

Marty Weiner: Jam. I was almost there. I who's wait, wait, who's number two?

Zach: you know, I'd have

Marty Weiner: The other guy.

Zach: I'd have to look up people who are not you who were the CTO,

Christopher: AHHHH

Zach: and with Marty Wiener, who is who's some other guy.

Marty Weiner: Some some other guy.

Zach: all right, so let's get into it. Big story this week that was going wild on Twitter was that large frontier models can now directly edit smaller models. There was a post on GitHub out of MIT by Takehi, Teo, Jong, and Khatab. quote We find that Frontier LLMs are surprisingly good at natively reading and writing neural network weights well past what we'd expect from humans or conventional algorithms. This might be a big deal for both optimization and interpretability. Is that true, Marty?

Marty Weiner: So I I gotta say that this to me is about as big as you can get. In fact, I didn't I mispredicted this completely. I thought we would need LLMs at sufficiently much larger sizes to be able to do this kind of work. But the surprising result of this paper is no, we don't. what essentially we have is you can use a frontier level LLM and you can directly read, interpret, and edit a neural network's raw weight tensors to to modify its behavior, which is insane. The the the implications are huge. to me this this has two pieces. One is this may have implications for massively improving the quality of of models in general. I think Chris you had some something on that. And then after that I I want to talk about the other big implication to me is alignment because now we have the

Christopher: Yeah I think

Marty Weiner: ability to edit alignment.

Christopher: Yeah, some extent this is actually this is this I think dovetails really well with a couple weeks ago we talked about the another another result where a researcher found they could actually start to attribute directly what each neuron in a neural network does in some sort

Marty Weiner: Yeah.

Christopher: of larger context. that just felt like it was gonna be like a giant, like almost archaeology project or like, you know, like AI psychology project to go through like kind of pick apart a model one one one one weight at a time. Here it's kind of the opposite. So like They gave examples with basically off-the-shelf frontier models. I think this was entirely done with GPT-6 Astra. And they were able to not only directly fine-tune and adjust smaller models that were open weights, but actually to build a harness around small versions of Quen. And for small versions of Quen, we're still talking about like the kind like billion billion node models and get it to do things that it couldn't normally do. So for example, they found that off-the-shelf Quen was not very good at doing math, one of those old, canards of you look, it can't even count the R's in Strawberry or or add two numbers. it went through and was actually able to get get Quen to go from like about a I think it was about a six percent to about a 48% on some of the math benchmarks. really drastic shifts over time. now the the the main critique that I've seen across this falls into kind of two camps. camp number one is for some of these models, as you can imagine, these are very small neural networks, 10 thousands of thousands of nodes in the neural network. And so it's a, I mean, this is just barely like able to tune like a multi-thousand node model. On the flip side of it, it's able to hand tune a multi thousand node model. I mean, like this

Zach: Yeah.

Christopher: is not like this is not like a this is I mean, like this is like a little bit beyond what a human would normally be able to do. the second thing that I've seen is like, well, it didn't technically actually modify the model. It wrote a bunch of Python code to modify the model. Which

Zach: Yeah.

Christopher: again is like that's literally how LLMs do complicated tasks at this

Zach: Yeah.

Christopher: point. Is like they don't I yeah, it's not it's not generally going in to do brain like it's it's not like it's using its hands virtually to do brain surgery. It built a tool to do brain surgery. So

Zach: Yeah. So so could I

Marty Weiner: Yeah.

Zach: could I like a ask to clarify so so to my mind the the reason this is cool is normally if you want to adjust the weights of a model you have to sort of go back and forth with it and making it run tests and that kind of thing. This is like Astra six looked at a giant list or a giant matrix of numbers and just decided which ones needed to be poked.

Christopher: Yep. Yep. Yeah. the the stare the standard way that you tr so so models are generally trained. And the thing about model training is that with a big asterisk, generally once you you you kind of have like do a one-shot train, and then the model you get is kind of the model you have. And then you can do you can do there's techniques for kind of like retraining and they usually there's something called fine-tuning. There's another thing called continuous P training CPT, and those are all involve adding additional data and making the model better in certain domains. So you can think of it almost like a, take a generalist model and almost give it a degree in some particular field. send it to night school and give it another, another, another, another skill. in this case, though, it's like it's pretty much just doing brain surgery. It's just adjusting the weights to make it so that it actually is better at tasks without actually doing a training round. Now you can ask the question of like, is this more efficient than doing training? You know, these are very small models, but I think like anything else, this is a new technique. and that's where the value comes in. This is this is a like completely orthogonal way to do this operation.

Zach: One thing we didn't talk about but I just popped into my head is d does this have implications for continual learning, which is still one of the big open problems in AI? if if you can

Christopher: I think this is like the it's like asking the question of what's the value of a newborn child? This feels like one of those projects of

Zach: Yeah.

Christopher: like, I I don't know where it heads, but it definitely heads in very exciting directions. the other bit that's interesting about this too is if you if you actually kind of like the the the the another another researcher came along and basically found that this is a feature that only exists in the most recent frontier models.

Zach: Yes.

Christopher: So they're able to demonstrate

Marty Weiner: Yeah.

Christopher: that it works in like it starts to appear as a capability. Right around the GPT five, six opus five boundary. And then all of a sudden, the the six variants and the five point five variants on on anthropic are actually pretty good at this. which breaks the question, is this because internal to the frontier labs, they are already doing this kind of operation and it was

Marty Weiner: Mm.

Christopher: just kind of like, or or is it emerging from all of the auto research they're doing to how to hope to try to try to get to RSI?

Zach: Yeah, see, that's what I was wondering is cause this looks like ooh, it's emergence, but it could also just be they trained it on the exact same behavior we're observing now, which is a little frightening maybe. because it means if this is a really big boost, they've already got it and have had it for a little while.

Christopher: this just feels like this does this definitely feels like a baby step, but it feels like a really exciting baby step, right? they I get the other day, yeah, we're talking about we're talking about thousands of of nodes or building complicated Python harnesses to modify more complicated models. But this just feels like a completely new technol new technology to deploy to be able to do this. So I think it does have heavy implications for things like RSI.

Marty Weiner: And I think the the other implication here that I mentioned at the top here is is the ability to adjust alignment in a more surgical way. naturally we have this problem that that the models, you you train this gig enormous model and you can push it in the training. And they've gotten really good at that, but this allows us to go in and find more nuanced problems of alignment and and tweak them. one example I ran into recently, I was talking to a guy who builds models at his company. And they keep running this problem that the models keep basically going crazy and deciding that all humans should die. That's that's what he said. Yeah. Uh-huh. Right, right.

Zach: Is that crazy? Yeah. I I don't disagree with that.

Christopher: that's like a it's a very it's a very rational decision to make, honestly, if you think about it. speaking sp speaking as we are on election day, I think it's actually not a terrible not a terrible choice.

Marty Weiner: Right, right. it's very

Zach: Mm.

Marty Weiner: i robot esque of it, right? so it gives us a really powerful tool to in a way control the frontier models potentially. and I know we're talking about a big model editing a much smaller model, but this is only the beginning. So if we can get I I don't know if we're gonna find some sort of an asymptote there, but if we can get to the point where GPT six edits GPT-7, seven edits eight. then we can keep the alignment where we humans want it.

Zach: Yeah. It does seem like there's an emerging picture that a way to solve alignment might be to create an asymmetry in in in the favor of the weaker model by giving it one directional access to the other one's weights. the the other thing I wanted to ask about. So one of the things that was really interesting about this is like when we say it adjusted the weights, it only adjusted a very small percent of them, and sometimes with huge results. And my my understanding is essentially it's like When you train the model, it's possible it's doing things in a stupid way in a number of cases. And so you if you can just untangle a little bit of it, you can get a big benefit.

Marty Weiner: Yeah, the the number was, let me get it exactly right here. Astra identifies an issue in its weights causing reasoning loops that Quinn's own model card warned about and fixed this by natively editing zero point zero one six percent of the weights. and

Zach: Yeah. So that's why I wonder if it's like it it figured out how to add, but you don't realize it's doing it in some alien, crazy, stupid way, and you can just say, Don't do that anymore.

Marty Weiner: I think so. And that was that they said that that's what they attributed to raising their math five hundred benchmark score from seven point nine to forty-eight point one percent.

Christopher: I think it's I think it's exciting. I I just think it I I I think for all the poo-pooing, I do think that it is a it's a novel technique. It's a novel technique. And so the question is with time, will this actually be something that is in certain domains better than doing something like fine tuning? I don't know, I don't know if I'm going too far afield here, but like does this allow a capability for models to kind of reach into their own brains and modify themselves live?

Marty Weiner: Mm-hmm.

Zach: Yeah, that's what I wondered about.

Christopher: That's I think one of the weird ones.

Zach: So so Alex Crenzel at at Google Research said, quote, pretty obvious next step here, self-improvement at runtime in the weights. But I suspect this isn't feasible for the current models due to the many orders of magnitude difference in size. But what what does that mean? That we you would be like using Chat GPT and it would reach into its own brain and improve itself somehow?

Christopher: It's just a bunch of weights

Marty Weiner: could

Christopher: on disk at the end of the day. So, if you if you really

Marty Weiner: Yeah.

Christopher: it I don't I don't know if you could it would require some interesting architecture because you can't necessarily do that to a running model. It might be tricky. but it's not it's not impossible. I mean you know, I I think we're I think what we're describing would be another capability to have almost like in in model literal memory. Like the kind of way the way your brain works is we don't have like a running

Marty Weiner: Mm-hmm.

Christopher: text file over on the side like most of these models do to kind of keep state. We just keep it in our weights and But we just don't know how to do that with neural networks. So the question is did the neural networks figure how to do it to themselves?

Zach: Yeah.

Marty Weiner: I I I don't know if this is too how far out this would be, but you could imagine one potential ability here would be to inject some knowledge. So add in capability of of solving chess.

Zach: So yeah, I guess I guess if you're trying to understand, so it would like you talk to the model that's bad at at chess and you say edit your own weight so you're better at chess. Is is is that the idea?

Marty Weiner: I think so.

Christopher: Yeah, here's a here's a benchmark. And it's actually what they do in the paper,

Zach: I see.

Christopher: which is like, here's a benchmark. Here is like and they do this with they do this actually coins. here's a benchmark for math. They show that coin is bad at math. And then they basically just tell Astro, make coin good at math.

Zach: Mm-hmm.

Christopher: And they just you can see this like there's a there's a great graph of over time it gradually train it's not training it, it's actually like adjusting it to be better. which is just wild.

Zach: Yeah. Or let's yeah.

Christopher: And again, it was all done in Python, so that makes it invalid somehow. It's like But I mean actually the neater parts are earlier on in this paper, like not to be labored but like legitimately it like gives it a neural network and just like that that they trained on different shapes. Like, you know, here's one's trained on circles, here's one's trained on squares, here's one's trained on triangles, and they like give it the weights and just like, what's this trained on? And it was able to tell like pretty quick, like it was able to walk through its reasoning, looking at the weights and be like, yeah, that's like a picture of a that's a circle.

Zach: So it's

Christopher: So it's almost like a Worshawk test. It's like a Worshawk test for L LLMs.

Zach: Yeah.

Marty Weiner: I wonder if the the big guys will be upset about this 'cause is this the next level of distillation?

Zach: That's what I wondered about. Yeah. Is like Quen's if if you take a a bigger model and it's close to the frontier and you can suddenly jack it up, do you

Christopher: It it definitely it definitely gets into the realm of well, there's okay, so for the there's no more closed sourcer even for models. I think that's why there's been so much comfort in shipping open weight models is that this assumption of it's actually hard to do the the training is expensive, even though we know that things like distillation and other techniques can be used to make models a lot faster than having to do train on all human knowledge. this is another way to say like yeah, if you open source a model, open source a op open source the weights, you're effectively giving you source code equivalent for the model. Or we're we're heading in that direction.

Zach: Cool. Well, let's move on to the big fake clo fake clawed code debacle. That's that was hard to say. so here's what happened is so apparently for a while cursor had something had an anthropic clawed code extension in its IDE marketplace. Apparently it was up for 11 hours, and they said up to 5,000 users downloaded it. and it was not from Anthropic, Anthropic confirmed this. There was a quote from Felix Riesberg on their technical staff. He said on X, we're working with Cursor on this incident, but it wasn't a compromise of anthropic systems or employee laptops. A malicious actor used access to Cursor's marketplace to push publish a tampered copy of the clawed code extension, which we didn't build or publish. So why why is why how did I don't understand how this even happens?

Christopher: Well, we talked about this. Okay, so it's funny. We we we we talked at the end of our discussion last week with Flea about supply chain attacks and how there aren't there haven't been as many as you'd expect. And here's a good example of a supply chain attack. So this is this is the the the problem that comes up here is okay, so cursor, which is a which is a an interface for doing software engineering with AI agents that's been around for a bit, got bought by SpaceX. they have what amounts to being their own internal App Store after a fashion of extensions you can you can add to cursor to make it more effective. And so you can install there's a codex extension, there's a cloud code extension, there's a whole bunch of there's the Python editing suite that makes it easier to look at Python code. in this particular case, one of the one of the longstanding tropes in anything like an app store is something like squatting on a name. And so if you if you happen to name something very similar. To something that is like the official version, you can convince some subset of people to click on the malicious one. And then all of a sudden it's like, and it could even work in a way that's very similar to the official one and just has an exploit in it. And it seems like what happened here is somebody just discovered they could just make a just make an extension called anthropic claude code, which is probably

Zach: Yeah.

Christopher: different from there's probably one that's just called Claude Code. And because of the way the search works, like people are searching for Claude and like it came up in the in the list, and some number of people just clicked on that one first. And of course, I'm sure there's some you ranking where it's like, as soon as people start clicking on it, it gets more boosts and it starts to show more, show up more. so I mean, to some extent, this is like you know, this this is a tale as old as time for app stores. I it's like it it goes

Marty Weiner: Yeah.

Christopher: back even to days when people would like, you know, phishing emails where it's not from Microsoft, it's from RN Microsoft, where it like you look at it and it's like it looks pretty close. this

Marty Weiner: Or that was the exact one. Yeah. Yeah.

Zach: Yeah.

Christopher: is the same sort of style of problem. and it's gonna be one of those things that like, you know, I'm sure Cursor doesn't have the same kind of internal DNA to avoid this kind of game, you know, gaming. And now

Marty Weiner: Yeah.

Christopher: they're gonna have to be much more careful about what they actually launch in their store because you could have a giant supply chain attack from this from this kind of

Zach: Well I that's what I wondered, 'cause this seems like something something obvious. You couldn't have security around this it it beforehand. It seems like something you would obviously want to search for.

Marty Weiner: a a lot of security is reactive. the shit, now we have to form a team. 'cause

Christopher: Yeah. Yeah. They did yeah, usually it starts off with they did what now? And then you have to

Marty Weiner: Yeah.

Christopher: work your way back to what's the feature you have to build. And 'cause like in this case, like

Marty Weiner: I I

Christopher: and there's no s there's no simple solution, right? It's like, you you you can't just you can't just void all ability to use the word quad in

Zach: I see.

Christopher: in extensions. And like they don't wanna go they're not gonna wanna build like a review flow or they're probably not gonna wanna rebuild a review flow, but it does create that sort of that sort of pressure. Yeah.

Marty Weiner: They probably have to. Yeah. this is why you have review flows, right? To watch for

Zach: Mm-hmm.

Marty Weiner: Apple has it, right? Otherwise you're gonna have a whole bunch of apps that are

Christopher: That's why Apple has it.

Marty Weiner: called, yeah, Google with three O's or

Zach: That's why yeah. That's why I wonder you're being the CEO and someone explains to you what happened and they're like, Well, they called it Claude. And

Marty Weiner: Yeah.

Zach: that was the trick. Just

Christopher: Yeah, it's the trick was the trick was they used they used words. They used their words on you on you. No, I like well the it's just one of the things too where it's like, yeah, they're technically violating trademark and copyright, but you it's not up to Claude to enforce that stuff generally. Or not that not

Zach: Yeah.

Christopher: to codex not cursor. Man, too many names.

Marty Weiner: W

Christopher: it is it's it's a tricky situation. It's one of those things over you what you want to do though is you probably want to have your set of like there's a bunch of these golden apps that should probably when you search for Claude, they should come up and like be so obviously the official labeled Claude app that no one would ever click on any of the other ones. But like I this happens all the time when you search it's this happens in the Google in the in the in the in in the both the Google and the Apple App Store. It's like you search for anything. You'll get this sponsored link at the top that's like a totally, not unrelated thing because somebody's buying that slot,

Marty Weiner: And by the way, this is not even a tech specific thing. I was just reminded, since it is election day today, this is happening exactly in Alaska. And and apparently it's a tale as old as time. There,

Zach: yes.

Marty Weiner: there is you know, there's there one of the Republicans is named Dan Sullivan. The apparently the story is the Democrats went along and got another person as an independent named Daniel

Zach: Another Dan Sullivan.

Marty Weiner: J. Sullivan Jr., so that people will get confused and vote for the wrong one. Supply chain attack on voting.

Zach: a it's a supply chain attack. They they they found a a different supply of Dan Sullivan's and I this that's right.

Marty Weiner: I think this probably goes back, thousands of years as long as democracy's existed,

Christopher: this is a this is

Marty Weiner: simply.

Christopher: a notorious this is a notorious thing that used to happen in Massachusetts all the time, too. It's like it's just like a lot

Marty Weiner: Yeah.

Christopher: lots of states do this when it's like they they they go and they just find that they find people with similar names and they want to crowd out a candidate and then just flood the ballot with like you it's like wait

Zach: God.

Christopher: which jimmy which Jimmy Sullivan am I supposed to vote for? why are there so many Jimmies? Okay.

Zach: my god. All right. the other big news was the so called tech sell off. Although I'm noticing the NASDAQ is up today, so maybe it's it's all over, but but but last yeah, the weekly

Marty Weiner: I know it's because

Christopher: It's micro crash. It was the micro crashes of the the annual mic the or the c weekly micro crash is over.

Zach: microcrash. Yeah, well, right.

Marty Weiner: C combined with the weekly, my God, the AI bubble, it's over. Everybody, let's talk about

Christopher: Yeah.

Marty Weiner: it on C N B C that's a daily hourly.

Zach: That that's a daily. yeah. So so to to to set some context, so on paper anyway, one point five trillion dollars disappeared from the stock market last week. and and I do feel like I don't know how how you guys feel, but there does seem to have been a kind of vibe shift. I think there was about two years of like the wallet just opens when an AI company points at it. And now it's starting to get to where maybe there's a little more there needs to be some delivery. so let me go through some numbers and stuff with y'all. So on October fifteenth. we got the first little bit of maybe bad news, which was anthropic who were seeking a two trillion dollar IPO. Apparently, of internally or I'm sorry, investors have lowered expectations to a mere 1.5 trillion. so that's ooh, ouch.

Marty Weiner: that's so low.

Christopher: Ugh. It's terrible.

Zach: but and then Reuters reported was October 19th, OpenAI wanted to raise $30 billion. And they want to do this before the anthropic IPO because they wanted to get that money before it goes somewhere else. And the the short version is they have not been able to do that. According to the Wall Street Journal, they only got eleven billion, which is, if you're worried about a bubble, that's a pretty bad sign. that they can't raise more capital or at least, they can eleven billion dollars is pretty good, but there was a time not long ago when open AI was oversubscribed for anything they wanted to do. so that's a bit worrisome.

Christopher: I I think also their I think their their target was around a one point four trillion dollar market cap in that case too. Which you gotta w you gotta wonder how

Zach: Yeah, and and and

Christopher: much that weighs in is like it's that's a pretty small slice of a pretty big number.

Zach: Right. Yeah. And they they have apparently lowered their expectation to 1.25 now. So they're also tamping it down

Christopher: That's number two, yeah.

Zach: a little. okay, but let's go through some numbers. So we had a bunch of tech earnings on October twenty eighth. Google reported negative cash flow, mostly due to AI spending. Their actual revenue was up a quarter, 25% year over year, at 125 billion a quarter. So revenue is doing great. Cloud revenue doubled from last year. It's at 30 billion alone. They were cash flow negative. Partially because of these AI spendings. Apparently, also they had unrealized losses because SpaceX stock was gone way down. there was also though this other concern that was interesting, which was that their advertising missed expectations. And so apparently there are concerns at Google that AI is cannibalizing search revenue, which seems like it had to happen eventually, right?

Marty Weiner: Yeah.

Christopher: if you're the the thing about the thing about the the AI overview is that it's the one case where Google has been aggressively pushing AI overviews at the top of their search results, which is pushing down ads. like pushing ads generally also below the fold, or at least like it's like just being a peak above the bottom of the screen, which has to have implications for for the efficacy of those ads. I mean, to some extent, like you know, the the running joke about Google for a very long time has been that it's an ads company that happens to run a search engine. And this is a good case of if they if they are if they're crowding out their own ads, which for a while, I don't I don't know if you all remember back in the pre-LLM days, it was becoming a joke that Google was turning into effectively a full page of ads with

Zach: Yes.

Christopher: maybe one or two squirrely actual real links slipped in between. now it's the opposite where it's like it's like you go there and you're getting effectively like an an an AI response, maybe with some ads below it. So it's not surprising they were able to, you they were they they're losing money on this.

Marty Weiner: I think the other issue too is the way that the world interacts with the internet now. It's it's far more I've got an agent, go get me a reservation rather than Google for it. or I I've got a question about physics, you know, like us nerds do. And and I go to Chat GPT instead of Google now. And so they used to be dominant. I I mean who who could beat Google going back about five years ago? So everybody would go to Google, everybody would see the ads. And then on top of that, I also wonder if they're losing ad revenue on just generic websites. if if if now I go to ChatGPT to get a recipe rather than going to the recipe site that's horrible and I have to go past twenty pages of awful ads, that's also lost revenue. of yeah ads.

Christopher: and you see this on you see this on third party non publishers of the SEO traffic is globally down and then increasingly drastically down because before it was already down because ads were crowding out the real legitimate search results. Now the ads are being crowded up by AIO reviews, which don't generally generate much in the way of inbound links. And so you kind of have this compounded effect of like the amount of the amount of traffic that Google is actually sending out to third parties is dropping precipitously with time.

Zach: Yeah.

Marty Weiner: Yeah. I wonder if

Zach: Well what's

Marty Weiner: the the way we're interacting with the internet is just fundamentally changing right now and we're seeing the f the beginning effects of it.

Zach: Wa w one one of the interesting aspects to this to me is this famous book, The Innovator's Dilemma. It talks about how pre legacy companies tend to lose to young upstarts. And part of that is because often doing the thing the upstart does would hurt your core business, right? So the classic case is Blockbuster, which had a Netflix before Netflix, they even had users, but they didn't push it because a lot of their money was things like late fees or people going into the store and buying candy bars and all this other stuff. And right.

Marty Weiner: Yeah, Kodak had a digital camera in nineteen seventy five, but that would have killed their film industry.

Zach: It would would have killed their business. But I I so I wonder, presumably a lot of people at Google have read this book and they know like, okay, even if we're killing search by providing these overviews, we have to compete. And so it's better to like burn down our core business than to like be assured of destruction eventually.

Marty Weiner: Well and I think they also have the challenge that they are out in the open. They have earnings reports and they have employees with stock portfolios that get affected the moment their stock dives and they are constrained in in a hundred ways that open AI and anthropic are not.

Christopher: I think also they take a look at the one of the headlines there is that cloud revenue up 103% to 30

Zach: Yes.

Christopher: billion. Thirty billion on a hundred and twenty-five is a significant chunk of the business now. I think this is this is in part just props to Thomas Curry and their CEO. but at the same time, it's like I guess how can you not make money in cloud right now? It's like it's hard to get it's hard to get machines. that you can you the GPU GPU rentals are through the roof. If anything, I've heard internally that it's very hard for Googlers to get access to their own hardware because it's just much but it's much more profitable to sell to sell it as part of Google Cloud.

Zach: Yeah. And so so the result of all that was that Google stock dropped double digits the next day. so but but it was which is fascinating because it it speaks to how much expectation there is and how worried people are about a bubble because a 25% revenue jump year over year is pretty solid. you know, most people

Marty Weiner: Yeah.

Zach: would say, but I think I think probably what's going on is the people investing this money want to see some returns and it's it's not here yet. Although Microsoft also did earn earnings, they had plus dollars.

Marty Weiner: Well.

Christopher: Wow. Are you allowed to do that as a business in the twenties?

Zach: yeah. S all right. It seems unwise.

Marty Weiner: Well, if you if if you don't seem to spend any money

Christopher: Yeah.

Marty Weiner: my do we see any AI coming out of Microsoft?

Zach: Well, yeah, right. that that is the problem. so yeah, so they their revenue was up almost 20% year over year, which by the way, just a a cool thing to me about all this, if you're worried about a bubble, these companies are still reporting blockbuster earnings quarter over quarter over quarter, even as the stock market kind of twiddles, which which hopefully means we're deflating a bit if there is a bubble. but but a similar story at Microsoft that Azure revenue was up forty-eight percent, right? and and their operating cash flow was way up. and they still spent a lot of Money on AI, just not the crazy volumes Google did. Amazon also reported their revenue was up a mere 12%, but similar story, AWS revenue was up 39%. Amazon, though, interestingly, they are deeply cash flow negative. they have negative cash flow of 42.4 billion over the last 12 months. And last year at the same

Marty Weiner: Wow.

Zach: time, they had a positive of 14. So they are spending. Crazily on infrastructure right now. and and the

Marty Weiner: Mm-hmm.

Zach: interesting thing is I think it's just they can't keep up with demand and they see money. and and and the question is, are there enough dollars to justify all this? But they also had a had a big loss in shares. a couple other small bits. Intel was way down. They're actually they they

Marty Weiner: they they tanked hard.

Christopher: They got tanked, yeah.

Zach: tanked hard. Yeah. on expectations of of

Marty Weiner: Yeah, it's a good thing we have a ten percent the US has a ten percent stake in them, right?

Zach: Yeah, yeah, that was a great move. Yeah, the US lost 12 billion dollars over that. That was cool. meta revenue up 26%, negative cash flow, same story. the the caught the the infrastructure companies are are getting hit pretty hard. and then there's Reddit. Chris, what do you think about

Marty Weiner: Ha ha.

Christopher: I like I like the alien. Very interesting.

Marty Weiner: Interesting. Hey, hey, hey, hey, hey, hey. You're

Zach: Ha ha ha.

Marty Weiner: s you're saying too much.

Zach: So, so yeah, so Reddit is an interesting story. So the the big thing to know is the stock tanked really hard. It's down 62% year to day, down almost two-thirds this year. 53% revenue increase year over year, mostly through advertising. users are

Marty Weiner: Yeah, that's incredible.

Zach: up 12%. daily active unique US users were down a little bit, but still they're expecting revenue to go up for Q4. Why are they going down? So we have a quote from Steve Hoffman on the earnings scroll. Quote The reality is the AI mode does not dry trap drive traffic. That is to say, the thing all the little recipe sites are experiencing, Reddit is is is facing too. Because, you know, when you use Google, a great way to get info was to add the word Reddit, you know, if you wanted to know about stuff. And that's a big source of their ad revenue, is that flow through traffic. So one of the big things they've done now is make it so the public has to sign up before they can see Reddit, which, you know. has you know, there's a long tradition that that was not the case. And they they're just basically saying they've been forced to do it. and so this is like, go ahead.

Marty Weiner: Well it's it's time for your yearly Redditors get really upset over some issue. And and

Zach: Yeah, right.

Marty Weiner: and Reddit's ending. It's gonna end in a month, it's over. The good times are gone. Yeah, dig. Dig is

Zach: I'm I'm switching to dig. I'm I'm going, yeah. no, but but so I, you know, this is in the grand scheme of things, this is a smaller thing, but it it has to me really interesting implications. Cause like if what AI does is it scrapes anything that you leave public on the internet, there's a real push to leave fewer things public. you know, so it was like we've already seen this. I I actually I don't know if you guys have tried this. I wanted to print out a recipe, which I I do from time to time, partially because I don't want to deal with the ads and I want to make notes and stuff. all the websites now require you to sign up for a newsletter to do that. And so it's like I I feel like it's it's depressing, the 1990s dream of the internet was this big public space where everything was shared. And now it's gonna everything's gonna be behind a paywall or at least a sign up because otherwise AI can scrape it. So it's on the one hand, AI is this amazing machine for getting information, but on the other hand, it's not gonna be able to access anything online in a in another year or two.

Christopher: the the tricky part too is that if you if you think about this from the standpoint of a publisher, there's actually the the extreme case I've seen here is if you look at there's a graph of stack overflows usage over

Marty Weiner: yeah.

Christopher: the last like you know basically since the beginning.

Zach: Yeah.

Christopher: And it's down to like it's it's it's it's down to just drugs of traffic. No one's asking questions anymore because instead it's easier to ask your LM. this is a problem for publishers, which is the the the the the the cornerstone of the internet for The last 20 years has been this kind of ad ad cycle, right? Which is that you get lead gen from other places like Google. And that means that brings people to your to people to your front door and you get ads. And the ads are basically about how you pay the bills. And then everyone makes everyone's happy in that situation. Now you have a situation where, well, agents can scrape that content. but that means that they're gonna be basically distilling it down and reserving up the answers without actually sending any leads to your front door. Which means the ads no longer are as effective because no one's coming to your tem to your place anymore. And so but there's not an all term business model here, right? It's like there's no money that was transacted in that. And so it's very it's gonna be it's gonna be dicey to figure out what is the future of that cycle.

Zach: Yeah.

Marty Weiner: I think it gets even worse. it gets worse in a in another way. if you're open table and people are using agents now to interact with you, like Instinct and Muse and and the other usual suspects, you are now saying, Hey, open hey instinct, please go get me a table at this restaurant, and instinct goes and it interacts with open table, and you got a table. You don't go to Open Table and Open Table gets all their upsales of hey, did you know about this restaurant? And and get a free burrito at at Taco Bell. So they're losing that revenue and they can't do anything about it. if you're Amazon, yes, you can go block Muse, but not if you're anything I think below that. You're you're like people will stop using Open Table if their if their agents can't get a table with them.

Zach: Yeah. Well, or maybe you don't even need open table, right? Maybe the agent, talks to another agent, right? at some point, we don't need software anymore. We just have agents talking to agents. w

Marty Weiner: Right, right.

Zach: from an economic standpoint, it's also like these companies where a lot of the money would go eventually, these AI companies, do not have large staff. they have a very small number of really well paid people who are very good at using AI. And that's a little worrisome of you would, you know, with all this stuff happening, if a bunch of stuff gets automated, the money is still there. It's just a question of how it's distributed. And and if if a lot of it's going to these companies that have three employees, that's a little worrisome.

Marty Weiner: Well, I think my prediction is a lot of these companies are gonna have what I've called for a while the the new Kodak moment. The they don't keep up with the times and then they they tank hard.

Zach: Yeah. okay. The last thing we want to talk about on the the sell off stuff. So on November 2nd, Anthropic put out new financials, which were very interesting. So Q three revenue, they reported 17 billion, which is as much as they had in the first half of the year. And it's 47% growth from the previous quarter.

Marty Weiner: Yeah.

Zach: absolutely bonkers numbers. You know, the 10x more than 10x in a year. they had a small operating loss, about 500 million. And that was after they made profit in in Q2. So pretty crazy. a lot of people were citing they have future purchase commitments of of close to a trillion dollars. But you know, that's spread out over a number of years. And like, look, if you're making like a hundred billion dollars a year, you can have a lot of purchase commitments. so I, you know, I I I I've started to feel like the the the only real open question is is is like. Are people getting enough value out of companies like Anthropic and OpenAI that they can just start justifying this stuff? And and really, if Anthropic does one more 10X, which is a lot, but they just did it twice, then

Marty Weiner: That's a seventeen trillion dollar or what what

Zach: That's the thing, it's like it's it's like

Christopher: Yeah, it's it's it be it becomes the it becomes the company, the the the futures dystopian science fiction, there's just the company that you just get everything from. you don't need anybody else.

Zach: Right. Yeah. if they ten X one more time, they're amply the largest company ever in history. Right?

Marty Weiner: Do you think we're far from I know that there's plenty of competition. You have open AI and Anthropic, and and that is probably going to protect them from monopoly you know problems, but has the US faced such a large company? I mean, I I guess we've had a we've had a couple of these cases in the past. the maybe the RCAs of the twenties, standard oil.

Zach: Right. standard oil. Yeah.

Marty Weiner: do we have does the does this start to get into a national security problem?

Christopher: Standard Standard Oil traditionally would have been about a trillion dollar company. And Standard Oil Standard Oil

Zach: Yeah.

Marty Weiner: Wow. Okay.

Christopher: Standard Oil is treated as the canonical Uber monopoly of the eighteen hundreds. It's

Marty Weiner: huh.

Christopher: barely a it's barely in the top ten of market caps at that point.

Zach: Yeah, although I probably worth noting like Standard Oil would have had more pricing power than I think anthropic currently has. that that would be the scary thing. Yeah, yeah. So

Christopher: It had eighty it eighty it was ninety percent of the market. So it had it had yeah, it was a monopoly. It was a monopoly.

Zach: so you got it's it's interesting to imagine. You can imagine some world where Sam Altman manages to keep Amode in in house and the main player now is open AI. They could probably charge, ten times what they're charging now and people would have to pay up. But we're happily not in that world.

Marty Weiner: Mm-hmm. And they and almost certainly they would have gotten broken up. It might be the best thing for OpenAI that that Amadei left and started Anthropic.

Zach: It could could be. I have trouble imagining the current the current administration

Marty Weiner: I

Zach: doing that.

Marty Weiner: plus this crazy, yeah. there is that side, but I'm also thinking about just the crazy drive. to get employees, to get this great team building, I'm sure it'd be exciting with open AI, but to have a competitor that is at your neck the entire time, it

Zach: Yeah, yeah.

Marty Weiner: yeah, it must must push for a lot of innovation. I also wondered this changes the dynamic of of a number of other things. We talked about this in a previous podcast. That you know, I I remember when when OpenAI was starting starting to talk about their new chip, jalapeno at the time. I was like, God, that's that's a lot of money to put in that. But it's like, wait a minute, we're talking about raising 800 billion, making a chip is now peanuts, right?

Zach: Yeah.

Marty Weiner: it really changes the dynamic of of what these companies are capable of doing when we're talking a seven trillion dollar market cap.

Zach: Yeah.

Christopher: I in fairness Anthropics also setting up a a bio wet lab as an as an extracurricular activity just because they can. So

Marty Weiner: Mm-hmm. Yeah.

Zach: Yes. Yes.

Marty Weiner: Right, right.

Zach: Yes. let's keep rolling. So there's a bit of robotics news. so probably the biggest thing was that the White House on October twenty eighth put out a proclamation basically concerning the fact that the US imports a lot of robots and a lot of parts that go into making robots. As as a as a a bit of a trade geek, one of my favorite stats about the US is Is most imports overwhelmingly. I think it's two-thirds of all imports are actually inputs. They're not like stuff you buy. They're inputs into stuff you buy that we manufacture here. so I think the broad reaction to this, so the the White House, they didn't they didn't get too specific, but they like did an investigation into imports. There was talk of tariffs. I think there's a big pushback from the business community and frankly, from like basically all economists are pro trade. The ones that are not absolute free traders tend to be much more pro-free trade than an average person. but the you know, there was a legit concern about cyber risk, but but this just seemed to be really heavy handed. Was that your reaction?

Marty Weiner: Well, every time they do one of these tariff things, it it it really screws the market. and and I guess in theory,

Christopher: Chaos. Complete chaos.

Marty Weiner: complete chaos. because I I mean you've this even happened with some of the startups I've been helping. Like you suddenly want to order a PCB from PCB Way and you're like, Well, what are the tariffs? Okay. And I get the idea of a tariff being there to encourage in-house build, you know, it within the US, but you introduce it slowly.

Zach: Right, right, right.

Marty Weiner: yeah.

Christopher: Also, it's it's been unclear repeatedly if these tariffs are even legal. So they end up in this weird situation

Zach: Right.

Christopher: where they impose a tariff, and of course, businesses are have to have to collect it because they don't know if it's legal or not. And then the course case court case comes along and it's like, no, that wasn't actually a real tariff. that tariff wasn't legal. So now there's this weird limbo of like, who do how do we give the money back? Do we give the money back? There's businesses who decided not to. All it does is it just jacks the prices up. And then there's no there's no pressure release valve

Marty Weiner: And slows down innovation.

Christopher: and slows down innovation. I mean I can understand from the standpoint of like, you know, in the same way that there's always been this kind of fear about like, you know, don't import Chinese routers for business use because it's too easy to backdoor those and you don't want your bit your base infrastructure to like backdoorable. I can't

Marty Weiner: But but not a stepper motor.

Christopher: I can't quite see it with

Zach: Ha ha ha.

Christopher: I have a lot more trouble seeing it with a stepper motor. I mean, like can imagine like

Marty Weiner: Uh-huh.

Christopher: there's a future where they're you know, there's like two LMs are communicating entirely via stepper motor noise. That just seems like a

Marty Weiner: Ha ha.

Zach: Yeah.

Christopher: That'll be a paper next week at the rate we're going. it's a weird it's a weird side channel attack to break break in it turns on the baby stabber. but

Marty Weiner: Yeah.

Christopher: but

Zach: Yeah.

Christopher: but yeah, this just feels like another way to like I don't know. This just it feels like chaos. Feels like

Zach: Yeah. Yeah.

Christopher: not very well thought out chaos.

Zach: Well and then the other problem with chaos is it makes it hard to plan even if you don't know that there's a tariff on something. So you might be nervous about doing anything with China, even if it's if even if there's not going to end up being a tariff.

Christopher: it it probably also as a manufacturer, it actually probably encourages you to move your stuff overseas. Like get out of the country.

Zach: Right.

Marty Weiner: Mm-hmm.

Christopher: It's like if I can if I can't get supplies of like basic printed circuit boards in the US because it's a pain in the ass with terraps or it's more expensive, then yeah, I'll set up my fab line in like Canada or Mexico or somewhere else.

Zach: Right. Well exactly. And that's like like I said, the like most of the stuff we import is inputs into stuff we make. You know, it's it's like our manufacturing is downstream. the other thing I just just by the by, there's this this is well studied. Trade promotes peace between countries. People think it doesn't,

Marty Weiner: Yes.

Zach: and they'll be like, World War One happened and there was trade. It's like that's true. It's it's not perfect, but actually, you know, there there's pretty good data that countries that do a lot of trade are less marginally less likely to go to war.

Marty Weiner: Yeah, I I don't want to attack a country if my PlayStation 5 is at risk. seriously. Right.

Zach: Absolutely not. Yeah. Yeah. Yeah. Yeah. I love that that was your go to though. It's so what wasn't democracy. It was it was

Christopher: No t was it that was that joke about no no two it's like no no two countries with McDonald's have ever gone to war

Marty Weiner: Yeah yeah.

Christopher: with one another, I think was the was this was the I'm not sure it's true anymore, but that was a it was too much for your recurring, I yeah.

Zach: I I I that was true before Ukraine, unfortunately, but yeah, yeah.

Marty Weiner: Right.

Zach: Yeah. But yeah.

Marty Weiner: But I think it's an obvious reason. I I if you've got two countries heavily depending on each other for your for

Zach: Right.

Marty Weiner: your commerce, you're gonna be like, No, I don't wanna go in war and and screw up all my all my my econ. Yeah.

Zach: Right? Yeah. There's an argument that decolonization happened because, you know, it turns out you can just trade. It's it's kind of magical. But anyway, let's keep going on robots. So a little thing, Kroger reportedly now has an inventory robot. Marty, what did you just pulled up some mature content, Marty? Ooh, that's hot.

Marty Weiner: Well this is the mature content, because it was on shitty robots and maybe apparently that's I how how is this NSFW? Yeah, that is hot.

Christopher: Ha ha ha ha.

Marty Weiner: yeah, Kroger has a robot. We were talking last time, I think about or maybe it was two times ago about Kroger laying off three thousand people in corporate. it's corporate. it's it's a little off Main Street, but this now is getting more on Main Street.

Zach: Yeah, cool.

Marty Weiner: so I think it's supposed to be doing inventory checking.

Zach: And then Meta, everybody now, has a humanoid robot. and there's there's a quick demo. It's it's kinda cool. I would say it's not not the most impressive humanoid robot, but maybe that's the sign of the times that it's not impressive.

Marty Weiner: Yeah, but now it also could be one of those things where we're just on the precipice of a bunch of competition pushing us up just like we did with AI,

Zach: I it seems like the problem is that there's some threshold where these things are useful and we are just not there yet, but maybe we'll keep going and we'll find it. You you wanted to talk about the the vibe robotics experience I don't like saying vibe robotics, but yeah.

Marty Weiner: Yeah, I don't know what to call this. We're we gotta find a name for it that doesn't sound dirty. no, I just wanted to some of my own experiments And I'm thinking that we're on the precipice of being able to make robots with vibe coding, vibe CAD, vibe PCB, everything. And so I'll just give a little bit about what I learned I have found that. Both the latest models from l like Astra and Opus five five are quite excellent at making circuit boards now, at interacting on my computer to make a circuit board. In fact, I could show you one of them right now, including the the schematic. So it made this entire schematic and added a whole bunch of notes. I had a couple of people yell at me on R slash KiCad

Zach: Yeah.

Marty Weiner: saying, my god, we're gonna just turn off our brains now. I'm like, dude.

Zach: Yeah.

Marty Weiner: How short-sighted you have to be. the the next layer here is being able to build up a whole layer just like we did with vibe coding. So now I mean, this is not a particularly complicated circuit, but it selected the parts. it laid it out, it made the PCB, it's able to to build all of this with with you know coming back and forth with me, but I didn't touch a single you know LED here. And then it's able to comp compose it into an engineering drawing. This is for an IR receiver. and it's got all the pieces here, and you can actually take it apart and see all the bits inside of it. And I have not 3D printed it, that's what's up next. I had a little bit of feedback for it, but the reason I'm showing this is I think vibe coding is coming for robotics, and so that's gonna open up a huge amount of of high speed development and and community around robotics. and to get even a little crazier, there's this great company that I played with called Meshi, which I found that one place where anthropic and opening eye suck is in making organic shapes. but they're really good at or actually, Astra is really good at doing a sketch of or making a two-dimensional drawing that looks excellent. So this was made out of Astra. But it sucked at turning it into 3D. But meshi.ai took that single picture and it made this. And since

Zach: Yeah.

Marty Weiner: this is for a Halloween thing for next year, I I gave it a photo that looked like this. Here's the actual photo from Astra. And it constructed this, including the back is exactly perfect.

Christopher: wow.

Marty Weiner: So then

Zach: Yeah.

Marty Weiner: I hand that over to Opus 5.5. It cuts it apart, it adds in the mechanics, and you'll actually see it here how far this stuff is. Come along is insane. Here's the dog cut apart. There's a few problems, but it made all the mechanics for the mouth to open, including where the servo will go. And you can actually see the entry point here in the back here it made for me to insert the servo and all the three printed parts. It's insane. and so I I don't

Zach: So how how how much twiddling? was it a lot of back and forth or was it mostly one shotted?

Marty Weiner: A lot of back and forth, but but more along the lines of you know, here's what I want, here's the dog, go build it. It goes off for an hour, it comes back and I say, Hey, there's no way for me to get the servo in the head. It says, good point. Okay. And then it adds a cutout the the great part is it's also a great design partner. So I talk to it and I say, I'd like to be able to do this with an Arduino. I'd like to be able to use these kind of voltages. And we had a good, excellent discussion. I the same kind of discussion I would love to have with a, elite engineer about choosing, you know, the what kind of power we need and and what kind of parts I want to use for this. and so I don't know, it's as it's as good as talking to a a very experienced electrical and software engineer.

Zach: Do you think do you think the the robot people it seems like coders have basically all gotten sold on this stuff in in the last six months? I I guess robotics people are not quite there.

Marty Weiner: I'm seeing it shift. So I got yelled at on R slash KiCad,

Zach: Yeah.

Marty Weiner: which is the software I use. But in another area and someone posted their circuit board. you know, it's again the classic story of where do I post my content on Reddit so I don't get yelled at. I think they posted a circuit board to R slash accelerate, which is a bunch of AI enthusiasts. And in there, there was a healthy discussion from other people who know about PCBs and and how to make schematics. so I my prediction is. Same thing with vibe coding. If you're not into it, you're just gonna get left behind very, very soon.

Zach: Cool. let's move on to there was a pretty cool paper, a paper getting talked about a lot. So just to tee it up really quick. this is by Lacoon and Balastriero. Lacoon, you may know, he was a very important person in the early development of AI. He was the lead guy at Facebook, at meta for a while for AI, and eventually they let him go. And the feeling was he was moving too slowly on the important technology, which was LLMs. Because he was insisting on these world models and in particular something called JEPA, joint embedding predictive architecture, which tries to abstractly represent things and and and learn underlying structure. And there was a new paper out this week, and and Chris is gonna be the one to explain it.

Christopher: yeah, so this this is interesting paper in that the the question that they're trying to to get to is if you were to take say two different AI models, train one only on pictures and have a a second one that's trained only on text, but they're the the question is would would those two models have the same understanding of the world? Right. And so so the the issue is is almost like the the the analogy they make is with Plato's cave. So Plato's cave is is this kind of like Very, very old notion of the re the representation of the world can be seen as the shadow on the walls of the cave. we it's it's a whole it's a whole long diatribe about the the meaning of a soul, but also it boils down to do LLMs generate the same shadows on the wall of the real world independent of how they're trained on the real world? and the answer in this case seems to be for certain types of models like Jeppa's, the answer is yes. so if you were to kind of take apart How the neural network works, it builds basically the same representation of reality. So the analogy would be like if you had a car crash and you had two witnesses on different street corners describing to you what what happened during the car crash, there's almost like an abstracted notion of the red car hit the blue car, right? But you might get completely different text, completely different stories from the two witnesses, but there's an underlying representation which would the same. The LL, sorry, the the neural networks here generate the same representation of Of that reality, which is interesting. in fact, originally it was a hypothesis. Now they've basically gotten to the point of they treat it as a theorem that this is the case. now the the kind of punchline here is this applies to JEPAs, it doesn't apply to LLMs. So for LLMs, they basically create this notion of they have this notion of like for for LLMs, all that LLM is doing, all the LM is doing, is it's a token predictor.

Marty Weiner: Mm-hmm.

Christopher: It's predicting the next token in the series. And so for the LLM, it's like the more extreme case of just asking one of the witnesses what happened. And having to predict what the next word the witness says is. That doesn't necessarily mean that it has a world model built into it. It means that it's able to predict the document. And then maybe if you squint at it and you say across all the documents, across all of the all of the world, it'll still be able to distill out some ab some abstract truth around how the world works. But that's not as clearly proven as it is with as currently with Jeppas.

Zach: Interesting. And so so I guess the the the question is I I guess the debate would be a lot of people do think LLMs do a good job of representing models by abstracting concepts, but but I guess well I I guess what's your reaction? Do you feel like JEPA is gonna matter or is it just gonna get swamped by LLMs?

Christopher: Jeppa, Jeppa is a very is is a class of models that is similar to I think it's the same class as Bert, which was the kind of state of the art going back about six or seven years in the ML space. they don't actually do text generation. Jeppas are more like they're more like classifiers. So they're gonna be much better at doing things like you give them a bunch of inputs and they're able to they're able to put a point in a space and say, this is this kind of thing. Right. This is like a it's it identifies it. And so generally when you make a JEPA useful, you still have to translate that representation in a vector space to some sort of an output, whether it be text or whether it be like an image. And so there's another model generally deployed to actually make it useful to to the likes of us. it's interesting though that they're able to demonstrate that the underlying representation is the same. this almost gets into like for the theme of the last couple of weeks

Marty Weiner: Mm-hmm.

Christopher: around this intelligibility of the neural network. The fact that they do converge on the same representation with two different inputs, one of them being text, one of them being an image with the same subject, is very interesting. It it does actually demonstrate that there is some notion of the world that's being baked into the into the neural network. I will say though, the one caveat here is that I think as you as you teed it up, Zach, is like Lacoon is a sort of like I wouldn't quite call him an LLM skeptic, but his his belief is that LLMs are kind of a dead end and that Jeppas are kind of like the the cool stuff.

Marty Weiner: Hmm.

Christopher: And so we actually has a company that's mostly focused on Jeppas. And so it is a little bit of one of those guy with company says that his thing is the best

Marty Weiner: Ha ha.

Christopher: in a in a research article. it

Marty Weiner: huh.

Christopher: isn't to say that this isn't like a I this again, this is another one of those getting a window into how do these things work. you know, for a very long time we've almost treated them like these are kind like alien brains that we're dissecting. it's almost the point now we should we should create like a LLM psychology section or a psychiatry section every week for this for this podcast because There's a one of these results now coming almost every week.

Zach: Yeah. So one of the big claims I saw was that, the the claim is that humans are much better at inference than L L Ms and the question is why. There's a claim that JEPA style models are gonna be much better at inference. And I I guess I don't really understand why.

Christopher: I I think the the the assumption is because they have a because they have a world model, they're much more grounded necessarily in reality. But that doesn't mean the JEPA's gonna the JEPAs aren't gonna generate text, right? The JEPAs aren't gonna you're not gonna talk to a Jeppa. You're gonna probably have a JEPA that augments an LLM or potentially that there's some additional transformer on the way out that converts the Jeppa into actually something useful.

Zach: Interesting.

Marty Weiner: I wonder on on the other side too, is even if there is a model like Jeppah's that's a different kind of paradigm from LLM's, it reminds me of why we are still building our our CPUs and GPUs out of silicon, even though in theory there's other kinds of of materials that might be better. But the engineering that has gone into building on top of silicon is just so vastly far along, and we can make so much progress that even if something better came along. It's gonna take a lot of effort for it to even get close to what we can do on top of silicon. And I'm wondering if that's the same case here. The there's so much research going into L LLMs, there's so much improvement that that for anything to stand up against them is is just gonna be a tall order.

Christopher: there is an underlying under an undercurrent of that that LLMs are kind of a dead end in this in in this in this research, which I mean, like they could be a dead end, but like not all all all breakthroughs that feel exponential invariably turn into some sort of a sigmoid at some point, right? Like, you know, not there are very

Marty Weiner: huh.

Christopher: few things that like most things are not exponential. There's always some limit that kind of comes in. But the question is like how much transformation has happened by that.

Marty Weiner: Uh-huh.

Christopher: By that be before you get to the sigmoid. And it seems like LMs are kinda doing that. Yeah, exactly.

Marty Weiner: Is that is that limit far beyond our right. And is that limit way far beyond anything we can comprehend? In which case we're it's a solved problem.

Christopher: A hundred percent. A hundred percent.

Zach: Well, let's move on to get to the quick news. I want to jam through some some of the fun stuff this week. it's probably indicative of something that we didn't even put this at the top, but Gemini for carbon. is that's

Marty Weiner: It's out.

Zach: yeah, well, it's not out. It's it's as usual for Google, it is declared. but but

Christopher: Outish. Yeah.

Zach: almost nobody has access to by the way, Chris and I were talking about this. So Argon, I think, is 18 on the periodic table. It's a noble gas. if you were gonna go for the next noble gas, it should have been Krypton. Carbon is a non-metallic and it's it's just like what what are they what are they doing? Like this they're not nerding properly.

Marty Weiner: Choose cats like everybody else.

Christopher: Well, what and also okay, let me just let me just let me just rantic Google. So we yeah, okay, so Google

Zach: Yeah.

Christopher: Google seems to be taking this tack of once every roughly six months, they do a drop of a new model, which completely blows all of the benchmarks out of the water. And generally also it's like a big it's like a big shot across the bow against all the foundational models. And then they proceed to not release the damn thing.

Zach: Yeah.

Christopher: And so it's like it's like good luck finding good luck getting tokens to use against either Argon or Carbon. And they're you they're coaching it under the same it's like fable grade, therefore you can't just or mythos grade, you can't just release it, it's like you have to be safe about it. It's like a likely story, buddy. Like it's kind of like you know, it feels a little this feels a little little funny funky.

Zach: Yeah. Yeah. I feel like I feel like Anthropic does and Anthropic will be like, hey, new model that might kill us all. I don't we're very worried about it. Do your best. And it's like mind blowing. Yeah. Yeah.

Christopher: Give it a try. Yeah, here you go.

Zach: yeah. So anyway, if it's if it's, you know, if the sets are real, it's quite good. It's nothing like mind blowing compared to the other frontier models, but you know, maybe Google back in the game if they get this out soon. Next thing

Marty Weiner: They can release it.

Zach: there's a quick thing from Google called Google Sleep. I don't have much to say, but it was an interesting paper. And essentially, they're trying to get to solving this problem of continual learning, which has been an issue since the beginning, which is that if you if you try to get models to update their own weights, they often have problems. They start forgetting stuff. And so this is an idea modeled on human sleep where the the model essentially goes and uses this extra set of memory to do something they call dreaming, where it comes up with synthetic practice examples and then updates based on them. and they found it it did a bit better on some tasks. But I don't know. I just thought it was a interesting paper out of Google that, another attempt to model what these machines do based on what we do. another big thing from Google, which everyone was excited about, I'm not sure what the implications are, but was Genie 4, which was sort of like on the fly video games. You load up this video game, you can even type stuff like, I want to see, you know, a dolphin wearing tights walk by and it would suddenly appear. and it's it's a fairly consistent world. But but Marty, why should anyone care about this?

Marty Weiner: we're seeing a huge explosion in games being made, where the LLM constructs the actual code for the game and you run the game. This, back to my prediction, I probably say every podcast, that J Grand Theft Auto seven will come out before Grand Theft Auto six. this is a world where rather than make the code and the geometry, it is just hallucinating video at you while you actually you can see them actually pushing buttons and moving the mouse around to to go around the this world. and that's that's pretty amazing. There's a lot of you know, a lot of things you could do with that. and people have actually said, you know, do a Minecraft, do a Grand Theft Auto, you know, like game. there are some big challenges with it. One of them is that if you can imagine one of the jobs here is just to predict the next frame. Well, one of the classic problems is you look in one direction, you look to the right, you come back, it's all different because it's just predicting the next frame. but they are getting better at storing at least some some notion of of of past drawings. But actually, that's just fine for a lot of stuff. You know, you can imagine a game where you're you're driving through a city never stopping, you don't ever have to see what's behind you. And you can construct that. So if this stuff ever comes out and is at all tractable price-wise, this will be an absolute game changer for I think how we consume content.

Zach: it seemed cool for me as a world model thing. maybe the the machine could think about stuff by running this. but I I I guess I understand like surely it would be easier on a CPU to run like a three D model that that you had designed in Opus than to like on the fly generate frames.

Marty Weiner: Yeah, but you have to generate it all. You have to get the lighting right. this allows just a free form, do whatever the hell you want world, rather than hey, go off for an hour and come back. So it it could be

Christopher: Yeah, I think this also has a nice, nice approach for open world, open world RPGs or an open world game

Marty Weiner: Yeah.

Christopher: where it's effectively you don't have to, as as a game designer, you don't have to go through and literally build every single surface and every single, pixel. And you you see this even in like with like, you the the canonical benches around things like Skyrim, where you know, Skyrim is like real it's a very rich, beautiful world, but you can start to see all the copy paste that are happening, right? Because like at the end of the day, like, you know, the humans are lazy and there's only so much time in the world. And so you're like all the caves kind of look the same. Whereas like in this kind of case, You could actually be building the map out as you're moving around. and you could just kind of give the give it sort of guardrails around like, you know, make a quest around this kind of thing. Like here's the goal of the quest and then have to figure out the details.

Zach: Yeah. And those speaking of games, so one of the big pieces of news out this week was that Claude Code now refuses to clone games.

Marty Weiner: Yeah, yeah, this this this is kind of a weird one. I don't I I don't know if they got yelled at by one of the big developing

Zach: Mm.

Marty Weiner: companies, but a lot of people have been using the LLMs lately to do a number of things. One is so someone said, Hey, when when the trailer for Grand Theft Auto six came out, they said, Hey, can you make me that? And it made it. I mean, obviously it's gonna be a much smaller, I mean, like point one percent of the game, but But that started becoming a fun sport. And then of course Steam started flow, there's tons of clones of Mario Kart and and the latest Zelda. And then people started doing what are called decomps and recomps, meaning you take your favorite game out there that you actually have the code for. So take Zelda and and you you put it on the computer and you say, Hey, you know, LLM, go break down the machine code into human readable code. And then recompile it. And so people actually now have all these games running natively on their computers. And I think Anthropic probably got spooked by IP infringement coming after them. Incidentally, I don't think we've seen the same thing from OpenAI, right?

Zach: yeah, I think that's right. It's yeah, that's a good point.

Christopher: but this is

Marty Weiner: So what you

Christopher: also that also lines

Marty Weiner: do is you s

Christopher: up with lines up with the way that like anthropic does tend to be a little bit more knee jerky in these situations. Like they tend to be the ones that make the change

Zach: Yeah.

Christopher: kind of more drastically. You know, they're they're the first ones that are gonna do watermarking on the requirement of the EU that they do it. they're the ones who, you know, they were the first ones to basically say, like, no, mythos is too scary to use. You should totally not use this, it's too dangerous. And then, you know, get the government involved. I mean, in this case, it

Marty Weiner: And then yeah.

Christopher: it does it does this is a weird situation too, because like I don't know where copyright law comes out in all this, right? Because to some extent, like Source code could definitely be copyrighted. I don't know if binaries can be. And therefore the question is is reverse engineer binaries that's a that's like time try a time tested way to operate is like you just take the output and you reverse engineer it. And that's not considered to

Marty Weiner: Well yeah.

Christopher: be copyright infringement because you're just taking the, maybe you get a copyright if you're like you happen to use Mickey Mouse or you make three circles that are suggestive, you can get in trouble, but that's about it.

Marty Weiner: There's another weird thing that I think could occur, which would be extremely wasteful. But imagine yes, you're not allowed to put out your recomp of Zelda Breath of the Wild so that everybody can play it on Windows, but you put out a single sentence and an instruction to copy binary over and say go. Everybody's version of Astro goes and spends, you know, a thousand dollars worth of credits. Well, I I

Zach: Yeah.

Marty Weiner: it's probably actually enough on the free model, and you you may not even need the the the money, and you've got your own copy. And so in that case, I don't know where the line is. you could say, yeah, GitHub, don't distribute my game, but can you tell people on Reddit not to say a sentence? you know, a prompt.

Zach: Yeah. let's keep going because we're we're getting out of time here. I want to get through a few more things. there was a a post by Anthropic reporting progress on accelerating drug development. So a lot of this stuff was over my head, but I I my impression was they were partially just helping find bottlenecks in and how drug discovery works, but also they were helping design new antibodies, which is pretty cool. it seems like essentially the process of drug discovery, as you might imagine, is pretty complicated. There's a cool quote on the blog that was quote, There was no single problem to solve. The bottlenecks are distributed all throughout the drug development process. so not too much to report yet, but I my impression is, the reason they're doing this is because it's a PR thing. I I'm sure they actually care about good results, but you it it's very hard to be like, Hey, all this data center stuff that people are worried about is okay because we'll cure cancer if you're not actually curing cancer. so that seems like the direction they would go.

Marty Weiner: Right. Well, there's also there's probably a lot of money there too, if they could start to crack some of these. the pharmaceutical industry is a massive industry.

Zach: Yeah, absolutely. Or even just providing help to get get things along. there's another cool paper. This is AI peer review. This is really neat. So there's a paper very quickly used Opus 4.8 and a paid review service, like humans hired to review, and then had them compared. So unsurprisingly, when AI evaluated peer review of of journal articles, AI always preferred AI. But even among the humans, 26 out of 30 times, they preferred the AI peer review.

Marty Weiner: boy.

Zach: so you know. human peer review is not always top quality, but this was actually an agency whose thing was doing peer review. so pretty striking finding I thought, 'cause I think this is an area where a lot of scientists I know would be like, There's no way we can trust AI to do this. This is a pure human thing. And it turns out, maybe, maybe we can

Marty Weiner: this is one of those spaces I keep thinking about how the AI will be biased, but perhaps it's less biased than we are, or biased in a better way.

Zach: Yeah, so I mean I actually I have heard whispers in different scientific communities of maybe there should always be AI peer review of every paper before it goes to publication. Just let AI do a pass for minor bugs. You'd probably reduce, you know, basic mathematical error citations, that sort of thing.

Marty Weiner: Absolutely. Yeah. Right.

Zach: and there's there's a funny coda, just real quick, that that they there's a guy who prompted Opus 5.5 to review the paper about this stuff. And And Opus 5.5 was like, you know, you're you're you're there's a lot of limitations to this design and analysis, and and you're overstating your conclusions and blah blah blah. So maybe we'll we'll have recursive peer review.

Marty Weiner: meta meta. Yeah.

Zach: little note from culture. you remember Elon Musk hated the Odyssey because it was too woke. and he was like, I'm putting out my own Odyssey and it's gonna be better. Yes.

Marty Weiner: Mate with AI.

Christopher: His main compla To paraphrase his main complaint was that many of the many of the actors in the Odyssey didn't look sufficiently ancient Greek. There were

Zach: Yes, yeah. Which is it was just very upsetting, of course. And so he made an AI version. the the thing I noticed watching this was the complete through and through lack of any kind of taste whatsoever. that that was off putting. I mean, I would say what's funny is it it it looks like it was made by a child, other than like the you know, visually it looks great, right? I mean, it looks like really high quality AI, but like literally every scene is a dramatic declaration made at dusk. and it's

Marty Weiner: Yeah.

Zach: like you cannot make a movie this way.

Christopher: It reminded me of do you remember, do you guys remember maybe 15 years ago, one of the earlier CGI movies that were supposed to be real was like this Beowulf that was like

Zach: Yeah, Zemeck is do it, yeah.

Christopher: and it was like all these kind of like glassy eyed, like stiff mannequins that were, you know, like in hindsight, you should have predicted it was gonna come out like that. This is feels like the AI equivalent of that, where it's like, you know, you could have you could have made this like two or three years ago using like, you know, stable diffusion or mistral and just been like, you

Zach: Yeah.

Christopher: know, like look at this great thing I made. Now it's kinda like it just feels like Candy Valley and stiff.

Zach: Yeah.

Marty Weiner: Couldn't couldn't he have gone and gotten spent

Zach: Actors.

Marty Weiner: a bit of money on well, well, I didn't even mean that. yes, make it an AI, but but get an actual director, get someone like Mel Gibson

Zach: Right.

Marty Weiner: who probably has a similar worldview to you and

Zach: Right. Well, they had some people, but but I I just I was sh the writing was almost nonsensical. it was it was like

Marty Weiner: Yeah.

Zach: it was like one after other, like dramatic speeches that you w you didn't even know what the context was. And it was just like bizarre.

Marty Weiner: Well, yeah, see, this is a this is a problem I think we're running into. There there's all these really fun AI videos that we've been showing on the show, but a lot of them suffer from a lot of people not yet being you know, many people are not directors. There there is a craft to directing.

Zach: Right. Yes.

Marty Weiner: and there's a craft to to timing and and and all of that. and I don't have it, and a lot of people don't have it. We are starting

Zach: huh.

Marty Weiner: to see some people though on like the R slash AI videos who Well okay,

Zach: Yeah.

Marty Weiner: that this this might this guy could be a sleeping,

Zach: Yeah.

Marty Weiner: Scorsese.

Zach: I I think he he did a good job of proving the opposite of the point he wanted to make with with this.

Marty Weiner: Yeah.

Zach: but but let let's start closing out. So we you can always say that later. All right, let's just

Christopher: Maybe it's just maybe it's just satire. Maybe it's just satire. It's Pozzola.

Zach: do the funny and weird section. best news of my life, MC Hammer

Marty Weiner: Yes.

Zach: is an e effective accelerationist.

Marty Weiner: Yes, yes.

Zach: he I had no idea. He posts almost nonstop AI of himself. like sipping expensive alcohol around sexy robots. and then talking about stock news and sports AI MC hammer does. and it's just great. I have no problem with this. It's wonderful. the future should it is what we want it to be. That's right.

Marty Weiner: This is why AI was made. Yeah.

Zach: my God. I I couldn't believe it. I was I actually I I found this and then I was like surely this is not the real hammer. and no. Did you guys by the way, I was gonna ask I I I I posted about this and some people were like, back in the day, he was like a Silicon Valley investor guy you would go to for money. Did did you guys ever

Marty Weiner: Yes. Yeah. I missed so I think he was at a at one of the investor meetings for Pinterest. And unfortunately I missed it. That is true. Yeah. man, I missed meeting hammer.

Zach: And then there was a little controversy with Sam Altman, who lamented the fact that RAM is ever anywhere but data centers.

Marty Weiner: Yeah, please, sir.

Christopher: How dare we how dare we plebs? We plebs and our and our pithy RAM usage. Why?

Zach: I see.

Christopher: We're destroying the possible future of civilization by using RAM for playing Skyrim.

Zach: That's right. I think what was funny is too is

Marty Weiner: Yeah.

Zach: using sympathetic Sam Altman voice too. it's just a shame that I don't know what the future is gonna be like if regular people have RAM. If they have some more RAM.

Marty Weiner: so so s somebody made a somebody made a good meme of it. Please sir, may I have some old Ram?

Christopher: Yeah.

Zach: speaking of which, our last weird thing is the the for a minute on eBay there was an option to buy an AI supercomputer.

Marty Weiner: You guys all missed out on it. And for the low, low price of almost sixty million dollars. Yeah.

Christopher: my god.

Zach: Of of sixty million dollars, yeah.

Marty Weiner: And apparently it was only fifty dollars shipping. I I can't find those details. looks like

Zach: Yeah, that's nice. It looks like the seller has a good rating.

Christopher: Ha ha

Marty Weiner: no, actually free FedEx, home

Zach: Good.

Marty Weiner: delivery.

Christopher: Nice.

Marty Weiner: free pickup within 60 miles. So that's a good deal. Yeah.

Zach: That's that's nice. Yeah. All right. And we are gonna we are gonna close out on this amazing music video, LGTM, which has been on replay for Chris and I for about a week now. so this is I it is definitely

Christopher: This is unf this is my life. This is literally my life, this video. I think as a software engineer.

Zach: the funniest takedown of AI, and it is made by AI, which I don't know what to think about that. but anyway, this has been Last humans in the loop. We will see you next time. Please make sure if you enjoyed this or even if you didn't to hit the subscribe button. It helps us a lot. and and tell all your friends Thank you very much.