Anjney Midha's Plan to Radically Lower the Price of Compute

0:02 Bloomberg Audio Studios, Podcasts, Radio News.

0:17 Hello and welcome to another episode of the Odd Thoughts podcast.

0:20 I'm Tracy Alloway and I'm Joe.

0:22 Why isn't thal Joe.

0:23 We like to talk a lot about physical constraints, yeah, on this show, right, And this is one reason why AI is a really fascinating area for us right now, because there are a lot of physical constraints on what is ultimately the sort of ephemeral technology. And I think that the tension between those two things is really interesting, Right, Like you type a prompt into chat, GPT or claude or whatever, and it's the sort of like disembodied digital platform.

0:50 You don't necessarily think about the power usage, the real resources, the transformers that have to go into data centers to get compute.

0:58 The thing that I've been on my mind and lately and I've written about it and I plan to write more, is this idea that the canonical AI thought experiment is what happens if you tell an AI to make a lot of paper clips, and then it destroys the world because in the pursuit of marshaling all of the world's resources, it just turns everything into paper clips. Because it doesn't know, and.

1:20 I have to ask, is this canonical example? Is this based on your traumatic fear of Clippy?

1:26 No, but that is you know, it all comes back full circle. But what we're seeing in real life is that everything from access to electrical grids GPU is being the big example, energy, turbines, talent, and now even including residential real estate are being reproposed to make more and more advanced AI. And in the original paper clip thought experiment, the envision, or at least in one version, the philosopher Nicolas Bostrom, envisions the AI having exhausted all of the world's resources, then sending a probe into outer space to consume star inner to build.

2:00 More paper clips eat the universe.

2:02 And to this point we're even talking about going into outer space for data centers to build more AI. So every version of the thought experiment is being replicated, except to just more and more resources to build the AI by humans rather than paper clips by AI.

2:18 There's this other connected theme here. So we've talked before about how one of the reasons valuations seem to be getting insane in the market is because all of this activity is being driven by like this existential need to become number one in frontier models and this new technology.

2:36 And so if you.

2:37 Say you absolutely have to be the first to invent AGI, then you can justify any amount of spending on Earth, right, And so what we tend to see is like the biggest companies just keep getting bigger. They're the ones that can get resources for all this stuff.

2:51 And I think one of the most fascinating things right now is that, at least as of right now June fourth, twenty twenty six, the frontier models are really close to each other, right, Yeah, so and the four point a GPT five point five, like they're not that different. And one of the things I'm curious about is is there something inherent in market dynamics in this space that will always keep you know, whether it's not being able to distill results from another model and cause I steal them, whether it's information sharing among employees. Is there some inherent reason why we've seen the stability or could it be that at some point one lab just like breaks out and establishes permanent still possibility.

3:34 But like I am personally on the side of commodification, and everything just becomes kind of basic or whatever basically available. I know Okay, well I'm just kidding, all right, thank you Joe as a Joe. All right, that's a polite prompt to get to the guest. We do, in fact have the perfect guest. We're going to be speaking with, Anjinay Mitta. He is, of course, a former general partner at Anderson Horowitz, a Stanford University visiting scientist who teaches the viral AI lecture called Frontier Systems. Also one of the first guys to write a check for Anthropic is now the founder of a new company called AMPPBC. So thank you so much for coming on our thoughts on Thanks.

4:14 For having me. One correction, it's pronounced amp pbc, but that's everything else you got corect on the.

4:18 Intro would make sense, wouldn't it as an energy Yeah.

4:22 That, and just remind us the PBC is public benefit corporation's right, So you're doing this for the public benefit.

4:28 We're governed by a public benefit charter, which means everything we do has to follow our mission. We have a public charter mission. We are for profit in the same way Ben and Jerry's or OREI and then Tropic our public benefits. So we aim to make a healthy, modest amount of profits that can sustain our mission. But we are we have the flexibility to choose what that margin is.

4:48 Can I just start.

4:49 I want to establish your credentials. Although I feel like that very long list did a pretty good job. But writing the first check for Anthropic like tell us that kind of origin story because the anecdote that you hear is like twenty five vcs turned them away initially and you said yes.

5:05 It was a little bit of the other way around. I said yes. Then we tried to get another twenty five dcs to say us and I failed. It was a harrowing experience. It was a bit of a wake up call. It was late twenty twenty. I had just sold my last business. It was called Ubiquity six. It was a three D mapping business. An AI is an AI business that we had founded in twenty seventeen. And I felt like a failure at the time because you know, in San Francis. I was in San Francisco. I just as a big picture my life stories. I was born India.

5:36 I went to high school in Singapore, and I came over to college to the United States at Stanford for my undergraduate degree. And then when I arrived at campus in twenty eleven. Deep learning had just started taking over the world in Silicon Valley. Andre Karpathy was a computer science ta to Andrew Ing, who was one of the i would say modern sort of founding fathers of deep learning.

5:57 This idea that you can teach machines to think without having to give them prescriptive rules, and so I went into sort of machine I got swept up in that moment and started studying. A lot of my coursework was in machine learning. My primary department at Stanford was in bioinformatics, which was machine learning applied to healthcare. I got sidetracked to a venture firm called Clina Perkins for about four and a half years, where I got the chance to work for some of the great investors like John Dore and Mary Meeker. And then I left and started my own company. And as is the case in Silicon Valley, when you start, I mean I was twenty five. I went and raised about forty seven or two million dollars

6:34 from some of the usual suspects like Benchmark and Index and so on. I thought I was the coolest kid in town, and I got the bit out of me because we built this incredible technology, which is this AI system that could map any location in three D. And then the pandemic hit, and so location based mapping, three D mapping. The only thing you can control is how you react to what happens. And so I did feel for a moment like it was bad luck. And then you just have to pick up the pieces and make the best of it. So I did with my co founder, we we figured it out. It was it was a tough few years where we had to pivot the business, but we landed the plane. We essentially a lot of the distributed systems we'd built on the back end side

7:19 ended up being quite valuable. We sold that to a company called Discord, which is a chat up for gamers.

7:24 Lost Discord fans in there. Yeah, awesome our listeners.

7:29 So, you know, about a month after I sold the business, I got a call from some friends. We're running research at Opening Eye, and we'd all been you know, friends in the machine learning community in the area, and they said, on you know, we've trained a little model called GBT three and we think it's the best sense just a little model. Yeah, nobody really paid attention. They were like, no, nobody cares, but we think it's the best thing since sliced bread, and we want to leave and turn this into a standalone business. But you know, it'd be helpful to get some of your advice on how to do that.

7:55 And I couldn't really come on board full time at the time with them because I had to integrate my company to the acquire But I came on as their angel, and nights and weekends I worked with them on the business plan and who we should raise from that. You know, that company was anthropic. Dario and Tom and I started doing these weekly working sessions in early twenty twenty one, and yeah, I assumed that, you know, if we went and talked to a bunch of venture capitalists on Central Road, especially some of the ones who were involved in the biggest hits of the last decade before, that they would get it. These are the creators of GPD three. And they were like, we just don't get this. We've heard the Holy I story before. This whole general Intelligence thing

8:33 is a pipe dream. And it was painful. We tried to raise five hundred million dollars. We couldn't. We instead scraped toge about one hundred million, which I know sounds like a lot bet at the time was a rounding era compared to how much Google has spent on the same kind of systems. And it was all angels in that first round, a bunch of cats and dogs, all of us who believed in the mission. And then over the next eighteen months, Dario Tom and team put together a plan that we kind of worked shopped on, getting an Amazon involved as a strategic and that resulted in a four billion dollar compute and capital partnership that made me realize infrastructure, especially compute infrastructure, was just a key

9:12 requirement to create any kind of modern eye lab. And so since then I've spent the past five six years figure out how to unblock that compute bottleneck for research teams.

9:21 Amazing well obviously an incredibly well timed.

9:24 It just like emphasizes how much things have changed, right where like people are literally throwing money at like almost any model now versus like a few years ago, going like hi, agi, really know, Well, let.

9:37 Me ask you this question, because this is a very top of mind question for me, and we're going to we can skip around on the timeline here, but there are three labs that are seen as like genuinely at the frontier right now, and that is obviously deep mind within Google, Open AI and anthropic And then of course you know a lot of people say that the Chinese labs are very close, if not there, maybe they're a few months behind. Is this is there? You know when we think about like part of your mission is like you say, okay, a new lab should be able to get access to compute if you're really bright, like that shouldn't be the bottleneck. Does that imply therefore that you expect more labs to be able to were they to

10:21 have access to the compute also reached the frontier, and that there is something inherent about like this sort of seeming stability or parody that we see among frontier models.

10:32 So the answer to your first question is yes, there have many frontiers to be conquered, okay, pioneered, and this is not just one frontier. I think that's a fundamental misunderstanding people have about your frontier. They talk about the jagged frontier, jagged intelligence right in a poetic sense, in a historical sense, if you think about the wild West or the Western frontier, it wasn't just one frontier. There was a frontier of gold, and there was a frontier of genes. It turns out Levi's you know, it turned out to be a new modern behemoth of a company.

10:59 I mean, there were so many new businesses founded in the Industrial Revolution, and I think that's that's the reality. Is the software engineering frontier, which is where Anthropic is clearly leader, is one frontier.

11:10 Yeah.

11:11 I think the you know, chat frontier, the sort of consumer chat frontier is another frontier where Opening Eye has been a leader.

11:17 Arguably byte Edance is that the video frontier with seed Dance, right.

11:20 Absolutely, yeah, And so I think there's just many many frontiers to be conquered are pioneered. Rather, I think Anthropic is clearly a role model for the rest of the community on how to do it in an efficient way. They're you know, I think he more than five thousand people, and they've been able to put out state of the art models that you know, teams like Google, which have sixty thousand people, are close to but not yet quite there.

11:43 So I actually I don't really agree with your with your assessment that they're all at parity. If you use the models day in and day out, they're quite remarkably different in meaningful ways to the person with hands on the keyboard, you know, doing the engineering work. And I think those those differences reflect the folk of the teams.

12:01 What is the actual mission that the team working on that domain cares about day after day after day. So in the Stanford Class I teach, the first lecture was a breakdown of how frontier models are even created, and it's actually quite simple. The recipes is super simple. There's basically four steps this pre training, mid training, post training, and then what we call the continuous feedback loop. So pre training just is just says, hey, you collect a bunch of data from the Internet and train a model to be a generally good pattern recognition machine. You then do mid training, which is to say, in a particular domain that you really care about, you inject more capabilities.

12:41 So if you want this model to reason about science or math or physics, then you give it science or matter physics data and then you get a pretty good model that specialized in that domain. And then you deployed to the real world where you have people using it. And the context feedback, which is when the model is able to do a task well or not and you can verify whether that task was done correctly, gives the model the data it needs to keep improving on that task, on that distribution.

13:22 This is slightly tangential, but like, I give a lot of feedback to the models because Joe made me paranoid about the basilisk theory. So I want the models to appreciate me once they take over the world. But when you give them feedback, like if they spit out a wrong answer and you say that's wrong, they immediately apologize and fall over themselves to say that they're sorry. But then you ask them, like give me another output, or like would you do it again the same way, and they like they often say yes or they give like a very similar answer. They don't seem to be responding in real.

13:54 Time, correct.

13:54 So when I say feedback, I'm in a very specific kind of feedback, which I call verifiable feedback. So when you say that wasn't right or that was wrong, that's an opinion, okay. Verifiable feedback is when you can have as close to factual verification as possible.

14:08 The reason So what does that actually look like?

14:10 That's a great question. So let's take reason by example in two or three cases. In the case of software engineering, in the way software engineer is actually code is you write a piece of code and then you submit it to the main code base, and then you usually have a peer on your team review the code and approve it or reject it, and if it gets approved that that's the first step that's called a PR a pull request.

14:31 And if another human on your team that you trust approved it, that's one kind of verification of quality. And then two before that piece of code usually gets deployed to a production system, you have unit tests, and those are quite objective tests of is this code performing the function we need it to And if it passes both those tests, it's a verifiable piece of code that accomplished the goal. So in software engineering, the reason we've seen such a dramatic improvement and capabilities is that a lot of these labs are using feedback from that verification loop.

15:04 In the case of another lab I incubated called Periodic Labs, which we started a year ago, and you should come by sometime, we've got forty thousand square feet in mental Park where we've got AI models that are predicting new The goal is to try to find a room temperature superconductor, and so these models predicted.

15:20 I forgot about superconductor I forgot about that.

15:23 That was a fun summer.

15:25 Yes, this time we will verify the if we ever put something on you, and you will know it's not that's not going to be us. But the AI system predicts new materials candidates. Then we have robots that synthesize the new material in the lab and then use X ray diffraction machines to test whether the material has the properties they said it would. And that's verifiable feedback from reality, from physics, and then we pipe that data back into the training loop over and over again. That context feedback is very factually verifiable, and that's where progress is the fastest today because that that feed back doesn't result in the kind of hallucinations that you often experience with these models on more subjective desks. It's also, by the way,

16:07 why the models is a terriblest objective desk, a creative writing and sometimes they can get quite toxic, to be honest, if you get them don the wrong loop. I don't know if you've been using it as a therapy, bought and so on.

16:16 I have not.

16:17 Just for the right, that's great.

16:19 It did ask me to defy the laws of gravity at one point because I was trying to create something in my backyard and I was asking you how to do it, and it was like, then just set this up like a following away and I was like, that's not within the laws of physics. Whatever.

16:33 No good, Well, what's interesting And this is actually a trillion dollar question from just a very broad standpoint is, as you point out, even prior to AI, the field of coding had a very systematized approach to the feedback loops already yes, and so then it's like a I could sort of replicate that anyone who's done you vibe coding can see in the chain of thoughts sometimes that didn't work. Let me try this, that didn't work, let me try this. Most fields don't really have that. By and large, journalism doesn't have that. I mean, there are things, there are outputs that are better and worse, we don't really have that that sort of like formalized approach to the yes no. Does that just zooming out? To my mind,

17:15 that would apply that maybe at least to some extent, coding is a little bit special from a sort of white collar knowledge work that in terms of like is it going to be as good as say, I don't know sales or something like that, because it has a coding as a long history.

17:34 It's formulated and structure that structured pipeline.

17:37 Yeah, this is a great point. So where progress will be made most predictably is in parts of knowledge work, where the task is essentially a workflow that's fairly structured. Yeah, and so somebody who spends most of their day in putting cells into an Excel spreadsheet, well, that part of the job will get automated pretty fast because that's actually verifiable, and you know what, that's frankly often the most tedious part of the job anyway. So I'm quite excited to see that progress because I'm terrible a spreadsheets, and I think if we could free up more of my time and hopefully other people's time, to focus on the art of the spreadsheet, not the deediest.

18:13 Part of the entry, the entry and retrieval.

18:16 Yeah, yeah, exactly, you know. And in journalism, I think it's the same thing. There's so much craft that gets that goes into the verification of a story before it goes out that's not legible to the world. You know. I've had a chance to spend some time with some of the journalistic institutions of the barrier like Cade, Metz or b Als and at the Journal, and as you spend time with them you realize, I mean they're verifying every sentence that goes into each fact checking, right, so fact check. That's an example where I think we should be leaning on these tools and you should expect more progress and the parts then that will be more to borrow jagget frontier framing. There, that's we will be in

18:55 a regime of jagged frontier progress where wherever parts of workflows that are verifiable factually will essentially you'll see progress they're very predictably over the next few years. And consequently, wherever that progress that the workflows are not verifiable is actually where humans are going to shine. And I think that's where parts of the economy are. You're going to see extraordinary gains in the wages of humans who have creativity and craft that are not typically verifiable you know, through traditional objective means. Does that make sense?

19:28 Yeah, it does, and it dovetails with a lot of what we've been talking about on the show recently. Just going back to verifiable feedback. So, Okay, the model spits out something and you can check whether it's right or wrong. Is it important to understand how the model actually got to that answer, because we have discussions with like big bank CEOs who are using more AI, and their response to this question is always like, well, if we can put restrictions around the AI, if we make sure that it's like released into a sandbox before it's released into wider world, we're all set from a regulatory perspective, and regulators don't actually need to know what's in the black box model and how it's working. But like this seems

20:09 a bit concerning to me.

20:11 Yeah, No, I'm quite strongly opinionated about this one, which is that technical literacy should be non negotiable. It's the reason I spend so much time teaching this class at Stanford, putting it up online, and the idea of the Frontier system's classes that end to end. It's a full simple but first principles of breakdown of how these systems are built from scratch, from land PowerShell, like the energy, where do we get them the data centers? Then how do we train the models? And the final project of the class with the kids was actually the one person Frontier Lab, which is at the end they're creating their own models and so on. Because the idea is that a person with the right tools today can scale themselves infinitely, but

20:45 they need to know how to use the tools, what the limitations are, when to lean on them versus not. And I think this is a generalizable piece of technical literacy that all leaders should have. It's like saying, you know, I in the nineties, I imagine if if you knew you could use the Internet without really knowing how it worked, but you know, on the margins when like the page doesn't like refresh or you're like this this cookie thing is annoying me. Like over time, people who are more technically literate just realized sometimes you've got a debug, you know, the browser, and those of us who've learned over time to do knowledge work are more adapt at leaning on them versus not. Like just now, when I was trying

21:27 to get onto the Internet, I realized, okay, there's this you know, Wi Fi password, whatever, And then you don't end up relying on them in ways that they can't fulfill your need anyway. And what's a little bit more dangerous with these systems is because we tend to anthropomorphize them without the technical literacy that that I wish all leaders had about reasoning about how these systems were built.

21:49 What you end up doing is projecting out in your mind what the capabilities are in ways that are inaccurate. You project out their impact and society that are not accurate. You project out their business in a way that are not Actually, I mean in the very fact that when you started this conversation, I don't blame you for it.

22:04 You're like, there's three models at the frontier. Yeah, I'm like, well, which frontier and which three models? Because from where I'm sitting, there's like seventeen different frontiers right now. There's four different players in each one, and the businesses of all of them are kind of breathtaking. So I think that technical literacy should always for leaders be a basic requirement. And then if you're deploying your these systems at Goldman Sacks, you won't oversimplify and get tripped up later when you know, two years later, you realize half your employee base has been leaning on this like sandbox framing, when in reality, inside the sandbox they were doing all kinds of They were using the tools in ways that were prone to hallucination,

22:44 prone to risks, prompt injection, they were leaning on it in ways that were not informed in the appropriate ways. Is this making sense?

22:51 Yeah, Like at a minimum they would not be using it in the optimal way.

22:55 Right or relying too much on it. It's the can't outsource your understanding to a model. Can you can outsource your thinking? You can outsource part of the tedious workflows, but you can't outsource your understanding. Yeah. And if you keep thinking, if you say, if you create these simplistic frameworks of oh, here's a sandbox and this is safe, you have to you have to use that sandbox in the right way. Because if you say, well, now, everything that happens at the sandbox is totally fine. That the model says, use the spreadsheet. The spreadsheet is good, it's deployed in our servers, but you didn't actually check the spreadsheet and what went into the spreadsheet. And then the model actually understand the particular structure of the business, the

23:36 physics of the business that you're trying to model out. Then you've outsourced your understanding to it. Does that make sense?

23:41 Absolutely? Let's talk about AMP Yes. And because one, you know, you're never going to get the frontier in anything unless you have access to compute. It seems pretty obvious, and there are various arrangements for acquiring compute. You have companies building their own data centers, you have smaller labs, and maybe they use someone else's data centers or a neo cloud, right, et cetera. What are you building at AMP such that at least as part of this story, is trying to solve the compute bottleneck.

24:10 Specifically, yeah, we are. It's very simple what we're what we're doing at AMP. We're doing two things. We are trying to standardize the format for compute, which today is super fragmented. So in the history of infrastructure, if you look at whether it was the Industrial Revolution, the Internet streaming, there were usually formats of inputs that were quite heterogeneous.

24:33 They were fragmented, and then to unlock productivity you had to standardize a format. So in the case of electricity, until it was until ACDC was standardized, right, megawatts would just sit in stranded pockets around the United States being unused. And then once we standardized the format to ACDC, then the question was okay, great, now we've turned all these stranded dockets of electricity into one sort of interoperable universal format.

25:04 Now how do we distribute it to everybody who needs it? And we came up with this distribution layer in the United States called the grid. That's that's all we're doing.

25:11 So building a grid for compute.

25:13 Correct, we're standardizing. We're trying to standardize the compute layer today, different chip types, different manufacturers, different clouds. I mean, it's a complete mess. And if you're go ahead.

25:23 Say more about how we plan to do this, because we've talked before about you know, there are various people out there that want to create indices of compute futures potentially on compute and the issue that always comes.

25:36 Up is fungibility, right exactly. So we've got a couple of ways we solve the fung ability problem. This is a pretty thorny challenge. We solve it in two or three ways. The first is we have a system called the grid, which actually makes the compute fungible at a consumption layer. So under the hood, we have a bunch of different chip types, we support various different manufacturers, and there's a system that was built to do this already inside a little company called Google, and one of the technical leads on that on that project was called BORG internally at Google is my co founder, Sebastian Lobo. He was my roommate at Stanford fourteen years ago. He's engineering co founder, and we're building BORG for everybody else, which

26:14 is essentially a translation layer that says, no matter what the underlying chip type is, the machine learning researcher who's using the chip just has to worry about the workload and we handle everything else underneath the hood.

26:24 When you say system, is this hardware or software that's doing.

26:29 That, it's all software. Okay, Yeah, So we handle that translation layer in software, and it's a pretty gnarly challenge. But today we're able to do that in ways that improved utilization sometimes from fifty sixty percent at labs that we have incubated or on the grid to close to ninety five ninety six percent. At Google, the utilization is roughly ninety nine percent. When Sebastian arrived at Google, it was about sixty two percent. By the time he left, it was roughly at ninety nine percent at Google. If utilization is at ninety six percent, that's considered a major outage. Today, the average data center in the industry, in the ecosystem, in the independent ecosystem is running at less than seventy

27:04 percent utilization. The Colossus two, which is running in Memphis, Elon's five hundred thousand, five hundred thousand GB three hundreds was running at less than sixty percent note utilization and less than eleven percent MFU model flop utilization is how much of the chip is actually being used. So there's two kinds of utilization people care about in the data center.

27:23 First is how many chips are being used. That's the highest. That's just the most naive measure. If that number is not a ninety plus percent, no excuses. So you have the chips, they should at least at least be doing something. And then within the chip during how much of the chip is being used within a workload. That number is usually much lower.

27:42 I'm very intrigued by this latter point about that even if even like the chip itself may not be even used at full pacity. Because I see these numbers and you say, like a lab has like a we have two hundred chips, we've acquired eight hundred GPUs, et cetera. And when I see these headlines, I assumed that ali optimal utilization techniques must be so good. That you can infer someone's capabilities simply by how many and video GPUs they've acquired. But you're saying is that there is actually quite a bit of heterogeneity about the techniques and approaches to getting the most juice out.

28:23 Of ad chip. Yes, you have to measure what matters, and what matters is output. Okay, you're when anytime I start a new lab with the in the case of periodic labs, it was we started with Liam Fettis was the co creative chat Epte and Dostubbuk who led the physics teams a deep mind. And when we sat down and we planned out the company's roadmap, the most important thing to us to measure it was not the number of chips we had.

28:45 Yeah, it's the the eval what we call all this chip bragging. They're like, oh, we acquired it is just a sort of bravado. Yeah, all right, this is helpful.

28:54 You don't measure the inputs you should be No, I agree, But I'm.

28:57 Actually fascinated that, like there is a saw toware solution to what I perceived in my head as like a very physical constraint.

29:05 How does this actually work?

29:07 Like feel free to get technical here.

29:09 Like, I want to understand the system.

29:10 Yes, so let me give you the technological answer and the economic answer. The economic answer actually is a simple one reason about the way the compute business works today is primarily on the construct of the atomic unit of long term leases. So I'm a researcher, I need some compute. I show up to a compute provider and say hello, I would like some compute please, And the computer provider says, no problem. Here's you know, five hundred AMD chips or in video chips that you can lease from me on various time scales, and you've got to pay for twenty four seven. It's like leasing an apartment, and whether you use it or not, that's your problem. But it's two dollars fifty per hour, three dollars an hour. So instead

29:50 you take a long term lease. And now the cloud provider, the comput provider said great, I just booked revenue for the next two years that this guy rented. Yeah, Now what happens with that compute, whether it's used or not, is the researchers problem. They've outsourced that problem as a result of this wastage that we're talking about. And I'm happy to go into why it's hard for individual teams to utilize most of the capacity. The primary reason it's because research is spiky. It's hard to forecast, so you put over provision for your peak, not your baseload. Because what happens you're researching on these algorithms, and the minute like one is working, you go, guys, let's scale. We want to ship this thing, so let's improve, throw as

30:31 many chips at it, and then once we ship it, the needs go down. So between these spikes there's just huge pockets of unused compute. As a result, the effective price per hour that you're paying is closer to twenty five to twenty eight dollars, whereas the marketed rate that you think you're paying is two dollars fifty. Yeah, so that's spread due to waste. It is just insane. So from an economic perspective, that's the wastage, that's the dead loss.

31:00 Yeah right, okay, So now how do we from a technological perspective, how do we utilize that opportunity? Literally all we do is from a software perspective, we take all of that unutilized compute, no matter what format it is. It might be in video, it might be MD we love AMD, it might be some other chip, and we turn it into one one fungible resource and that we standard as a format on something we call grid credits, So researchers don't even need to think about what chip type is under the hood. You know, they're just paying what they need or what they use. And so from a fiducial perspective, I'm on seven boards as an investor, I get very excited when when teams switch from this sort of long term lease model where they're paying twenty

31:39 five twenty six dollars per GP hour now they're actually only paying the two dollars fifty that was marketed because everything they're not using gets reallocated to the grid and other research labs can use that resource.

32:05 This is the problem you're mostly solving for. Is the training part because they're training right training? Or is it both both?

32:13 Yeah, that's the So the beauty about having diverse types of compute on our grid is that once you make the resource fungible, you can do any workload. You just fill all the unutilized pockets with inference and then all the reservations with training.

32:28 Yeah, so can you explain, like why is it that every lab also seems interested right now in customized silicon, including Microsoft announcing a chip that says like, oh, our new ma I don't know Maya, I don't know how it's pronounced.

32:45 I believe we had class yesterday at Stanford and he pronounced it as Mai.

32:49 Okay, their new MAI model. And he's like, oh, we also have a new Maya two hundred chip or something that's optimized with it. Why is it that so many labs or lab I guess, feel impelled to like also design a chip that go along, goes along with the model, and long term is what you're doing saying like this really is not necessary to have that sort of model chip alignment.

33:13 Yeah, there's two technological reasons and two economic reasons.

33:18 Okay.

33:18 The first is from an economic perspective, about eighty cents of every dollar a lab spends today on their R and D flows to a chip provider like Nvidia. Okay, right, And so as a result, your margins are just super super rough. So from a unit economic perspective, you want more control over your margins. And therefore, when you look at your unit economics, you're going, wait, wait a minute, For every dollar we make, there's this massive chunk that's going to somebody else.

33:42 So instead of spending eighty cents to Nvidia, you spend seventy eight cents to t subs and keep that two cents for yourself.

33:50 Well, I think that the better our software gets, the more that margins should flow actually to the researcher. Okay, because that's where the value will be captured.

33:59 But like, but wait, sorry, you were going to say, what's the technical reason why they're trying to do optimal model chip alignment.

34:07 On the technical side, the primary reason is you want control over your supply chain because today in a compute well we've been in a compute capacity constrained world now for at least four or five years. But if you can't get the chips you need, you're not in control of your own supply chain. So you're dependent on computer allocations that the compute manufacturer thinks is optimal. Right. By the way, that's how it works at the foundry level today, TSMC gets to decide which compute provider. Compute provider's business grows or not, yeah, because they only have so much production capacity. And so the technological reason is you want supply chain independence. And so when you want economic independence,

34:42 unit economic independence, and you want supply chain independence. You want as much control over your own chip.

34:46 But Microsoft doesn't have a FAB.

34:48 That's not what I'm saying. What I'm saying is an inference for example, Yeah, Satcha would like more control over his unit economics, so he's making an inference chip, right, Because if you're dependent on a third party to give you the inference chip, okay, and you need and if you don't have an inferenceship, you can't sell more more product you want more?

35:05 Is it about having a predictable supply of jest for you rather than a predictable supply of okay?

35:11 Yes.

35:12 So there's a lot of discussion right now about more efficient model allocation. So this idea that like, you do not have to be using the latest model to ask like what the weather is going to be tomorrow or something like that. And you also don't want to blow through your entire one year token budget in the space of four months, as Uber apparently did. So the spikes in usage that you're seeing that allow you to do the system and you have grid credits, does some of that go away if people become smarter about which models they're actually using.

35:44 Okay, so there's an embedded assumption. I think I should tease a part in your question. Usage is different from the production of the model. So what's happening right in terms of the pipeline is you use the grid to produce the system, the model, and then the model produces tokens. If the end user is only using tokens, then as long as everybody we have enough diversity in the end user base using models hosted on the grid, things actually even out. Okay, that's cyclicality in the same way electricity in America evens out if you have enough scale at scale basically as part except when like.

36:20 There's a heat wave.

36:21 There's a heat wave exactly, so some of that infrastructure we are having to reboot. But you can think about AMP in the broadest sense as a utility company where what's called an independent system operator of the grid. So we don't own our own data centers, we don't own our own labs, but we coordinate the capacity needs across different parties and a sufficient scale that those usage patterns actually just gets evened out. Does that make sense?

36:45 Yeah, Well you are an investor, Well, you are an investor in an open Router I believe, which I think is an interesting company. Do you see seting aside AMP for a second. Do you think that there is a at this point still within say corporate a Mariera, a certain lack of savviness about knowing which model to route to for the query, and that there will be an improvement in learning within companies, within users so that you don't have these incidents for like massive token consumption, because perhaps everyone was using the wrong the Cadillac model and the Ford model would have been just as fine for that purpose.

37:22 Oh yeah, we're absolutely in the medieval ages of this technology. I think what will happen is increasingly, based on my conversations with corporate American leaders and corporate leaders across the world, they don't really care about the models. They don't care about the underlying model the technology. They just don't care.

37:40 It's too much complexity. We just want the work done. Can you guys please figure out how to get the work done in the cheapest way, in the most efficient way, in the most secure and trusted way. And increasingly what you'll find is that which particular model is helping you out in a particular task will just be abstracted you won't even think about that. It'll just be a companion and you're just going to talk to It will be a companion provided by a brand new trust, and under the hood, it might be using the two hundred different models to orchestrate your ask and over time that efficiency will get better and better and better and better. And that's why I just don't think there's only three frontier

38:15 models that are going to win. Well, it's going to be an ecosystem.

38:18 This is I know you don't want my take, Joe, but.

38:22 Coffee, I love you take.

38:26 I'll save it for the outro. Actually, on this note, we have seen some headlines recently. Obviously there's the Uber one about token spending and I think it was the CEO said he wasn't sure if the ROI was there on Uber's AI usage. And we've seen there was a good Vox article recently about a corporate reckoning with AI spend.

38:46 Since you're going out and talking to CEOs, do you see any like Has anything shifted in the past couple months or so in the way people are thinking about the return on this initial investment or the return on spending on tokens.

39:01 Yes, I think it's a Barbelle distribution. So there's two types of CEOs broadly speaking. The first is the CEOs who are using the tools themselves, and those folks are going, aha, I understand the jagged frontier. When they understand the jagged frontier, we talked about their strategies, their questions they ask me are completely different from the CEOs who are outsourcing their understanding.

39:26 They're not trying the tools. They're mostly asking their kids like, hey, kiddo, this chat GPT thing, Like it's good, right, and your kid is like, yeah, it's pretty good, dad, And then.

39:37 Kids think it's really dumb by the way.

39:39 Yeah, So that's the other thing, right, So the kids are super smart and they're using the tools. They're like, it's good at this thing, but not at that. So they understand the jagged frontier part.

39:48 You know what, they think I'm dumb for using it. They're like, Dad, like you're not doing anything smart.

39:53 You know.

39:54 They don't think the models are dumb.

39:55 They they might be exactly, they might be going the way you're using it exactly not optimal.

40:01 So so the you know what I'm saying is my kids are sick and they have no intent and they have no idea about anything, and they just think I'm dumb. That's the that just might be a generalizable that's really the I see.

40:13 Okay, well you can send them over to me anytime. I'm happy to be the fun uncle.

40:16 Yeah, that would be great. You can show them that actually this is fun to.

40:20 My wife and I am happy to host hert That's really what I'm trying to get it. It's it's the summer, right. We have two nieces in London and we call it Camp Mida Shan. My last name is Mida, my wife's name is Shan, and so you're you're welcome to send them to camp anytime.

40:34 That's amazing.

40:36 But that's that's the bifurcation. As leaders who are actually trying the tools out, they realize they're extraordinary at some things and not at others. And so the depending on whether you get it or not, or you're actually getting your hands dirty or not, I find the questions are completely different.

40:50 So this has been incredibly helpful conversation in terms of like understanding basically the problem of it's actually tons of money is being spent, and your thesis is that it's massively suboptimally used up and down the stack. You mentioned this, Okay, you get a credit et cetera. Like, do you actually see that being financialized in a way? I mean, okay, I'm I bought this capacity. I have a lot of un new time. I mean, don't always have a research idea that it's going to require a big model, run test I can resell that. Is that something that you see like something that genuinely resembles a financial market?

41:28 I hope not, because when you had when you add speculation to you know, production goods, it creates scarcity of a different kind, right, because then you have financial traders and markets trying to trade the speculative value of the asset, and that's going to hurt a lot of our research teams in technology. On the other hand, I think that creates a need for innovation inside of the research teams.

41:53 And so one of the one of the cooperating functions we have inside of our business is a forecasting capability where we have a team that's very similar to actually the kind of forecasting team you'd have inside of a hedge fund. We're constantly predicting demand and supply, and then we're actually procuring capacity in advance through call options. On compute clusters. But our needs are similar to the kind of internal trading desk you'd have inside of a large steel company right where they need to lock up iron ore and so on for their production needs. So I'm a big fan of efficient markets, and I'm trying to actively invest in and help entrepreneurs out and teams out who are trying to drive more efficiency in the service

42:31 of more productivity in science and engineering. I'm not that thrilled about the financialization of these products if it ultimately results in more speculation. Does that make sense?

42:42 Yeah, I'm just curious since you're tracking demand in that way, Like if you were going to describe the slope of demand right now versus say.

42:51 Like a year ago, is it steeper?

42:53 Is it starting to plateau perpendicular?

42:56 Wow?

42:56 Okay.

42:57 If you look at the compute prices of long term rentals over the last six months from between January and now, they're trading up du x. So we start, for example, for twenty twenty six, we started securing our capacity in January at these long term rates, we could resell that at a duo X markup if we wanted to.

43:18 Part of the reason that twenty twenty six has become just totally AI has consumed everyone's mind. I think is because people got very excited about Claude code specifically. But that was a breakthrough at the harness level, not the model level, right suddenly, like the really exciting like wow, this is just so fun, it's just so easy of a computer inside your computer. That was a harness breakthrough.

43:43 Do you see, like when you think about investment among AI labs, do you see any shift in allocation away from pure scaling and improving the model towards sort of like tooling and harnesses as a way to get more gu out of the models.

44:01 No, I'm sorry, I have to correct you there. It was not just a harness innovation. Those two things go hand in hand. It's a symphony of improvement between It's a dialectic between the model capability and the harness. That harness was designed specifically for the capabilities that the new model was going to have, and so when you design these things in the industry, we call this code design. Okay, so you have the harness designed side by side with the researcher who's designed the next generation capabilities in the model, and you get a little bit of visibility and where the model is going to be good because as I described earlier, the pipeline is actually quite predictable pre training, mid training, continuous feedback loop. Once you have that visibility,

44:39 you go, aha, we specifically want to improve the capabilities on this type of task. It's going to take us about three months to get there, start designing the harness for that improvement. By the time they show up, then you can have the harness assume that the model will be able to do X, y Z on its own, whereas ABC is going to need the third party tools.

45:01 So then the harness says, remember that three months ago, you were terrible at understanding a spreadsheet. Yeah, so then we had to go right, like, go use a third party tool to use a spreadsheet. In the last three months, what we've done is added the ability to actually reason about a spreadsheet in the model and the model not.

45:19 So now you don't need to use a third party spreadsheet, okay. And so then the harness gets updated to say, don't go out and use a third party spreadsheet, which, by the way, collapses the time required to do that task by like sometimes a minute to two minutes. Now suddenly I've improved the user experience, and that's when things really sing.

45:34 It's when both of those those parts, the model and the harness are co designed to create a symphony. Does that make sense? Yeah?

45:41 Absolutely, all right, Anjinay Mitta of AMP PBC, thank you so much for coming on od lots really appreciate it.

45:48 Thanks for having me.

45:49 And everyone go out and check out the Stanford Lecture series. It's on YouTube right.

45:52 It is CS one fifty three dot Stanford dot edu. Perfect.

45:55 I'll I have a big flight coming up, so I'll watch it.

45:57 Then you should download all the lectures. It's quite a few.

46:00 Thank you so much. ONNJ. That was fantastic.

46:02 That was great, all right, Joe, that was a great discussion.

46:17 Yeah, I should emphasize just how big a deal that lecture series actually is at Stanford, Like students are beating down the door basically to get into that. And if it's free on YouTube, you should definitely check it out.

46:29 I just want to establish that if I had given you the Avian simulation that I didn't want to hear your take, or be the idea that I would have wanted to hear Anna's take instead of yours, I want.

46:40 To hear your take no, it's fine, Joe. I realized that most listeners are here for the guest takes. I get it, but I thought his point about the jagged front frontier was an important one. And this idea that like maybe the future it's not going to be a winner takes all things. In terms of models, you're going to have a bunch of different models doing different things that might suit different companies. And also the idea that like a lot of companies aren't going to care about which specific model they're using, they just want the cheapest one that basically gets the job done. In my mind, that sounds like more of a is commodified the yeah, a commodified market, right, rather than like, oh, people are

47:21 going to pay up for as you said, the Cadillac.

47:24 Well, so what I would say is, by in listening to Angina and amp is that people will want a commodified service, but that under the hood, I mean, this just sounds like what she's really trying to solve. And it's very interesting I as a user or a company by a commodified service. But under the hood, the commodity has an incredible amount of variety of models through which it can rouse sure some of which will be the Cadillac, some of it will some of it will be the current coffee cup.

47:56 Yeah, absolutely, But like my point is maybe in terms of valuations, sure, right, Like if everyone is assuming that the Cadillac is going to be like the one that everyone is going to get, and the total available market the TAM infamously is like not just the world, but potentially the universe. Like that seems a stretch to.

48:17 Me totally, and just generally I thought it was super interesting and the idea this is we've done a couple episodes recently, specifically learning more about both chip level and box level optimizations, both how many chips you're using and how well you're using ad chip Definitely when more to.

48:34 Do on that.

48:35 It still blows my mind that this is a problem that can be solved with software rather than like something physical. You just come up with a way to efficiently allocate the compute. Yeah, because in my mind, like it's it's such a physical problem. And we've talked to you know, previous DEI market participants like Brandon McBee at Core. We've and they talk about like, oh, it's difficult to standardize because of the configurations of chip and things like that.

49:01 But if you could solve it just through a software system.

49:04 That's pretty crazy. I guess Google's already done it. Yeah, all right, shall we leave it there?

49:08 Let's save it there.

49:09 This has been another episode of the Odd Lots podcast. I'm Tracy Alloway. You can follow me at Tracy Alloway.

49:14 And I'm Jill Wisenthal. You can follow me at the Stalwart. Follow our guest A Mida at a Jena Midda, Follow our producers Carmen Rodriguez at Carmen armat Dash, Ol Bennett at Dashbod, Calebrooks at Calebrooks, and Kevin Lozano at Kevin Lloyd Lozano.

49:29 And for more Odd Lots content, you should check out our daily newsletter. You can find that at Bloomberg dot com forward slash odd Lots.

49:35 And you can shout about all of these topics twenty four to seven in our discord Discord dot gg slash od loots.

49:41 And if you enjoyed this conversation, then please leave a comment or like the video, or better yet, subscribe.

49:47 Thanks for listening in