🔬 An Oscar, Two Asteroids, and the Algorithm in Your sklearn: John Platt on AI for Science

Are you talking about introducing explicit priors that, you know, based upon some human intuition or maybe in this case, LLM intuition? When you talk about multiple hypothesis testing, right, there's predictive models and there's descriptive models. A predictive model is like, let, let's say you just have a, you have some inputs and you have some outputs, and you just-- I just want to build a piece of code that tries to just have the lowest error rate on some dataset, statistical model.

A descriptive model is actually what science is trying to get to, which is okay, it should be able to extrapolate because it has sort of the physics or the actual some description of reality that's captured within it, and then you can use it to extrapolate. Yes, Newton thought of apples and gravity, but gravity isn't actually about apple, right? If you'd taken a seventeenth century machine learning model, like, "Oh, apples will fall, but how about planets?"

You know, "I don't know. I have no data about planets, so who knows what they do," right? The distinction between those is a little bit blurry, right? Because when a physicist or scientist comes, they use their intuition or maybe even more than intuition, like essentially there's maybe a, a solid pile of facts that they know about the world, and then they make sure that whatever model they build is sort of consistent with what's known.

Welcome to Latent Space Science. Um, I'm Brandon, uh, joined by my co-host, RJ. Uh, it's a pleasure to have John Platt, you know, with us today. John is a Google fellow and head of Applied Science at Google Research. He has really a fun, like, background. He, I guess you described yourself when we were talking a few minutes ago as a mega nerd. No, giga nerd. A giga nerd.

Giga nerd, get excited and absolutely everything, and, uh, it really, it really shows. Yeah, you was-- Correct me if I'm wrong about any of this stuff, but, um, so you started college at fourteen- Mm-hmm... and started your PhD at eighteen- Yes... at, at Caltech. You're, you were advised or co-advised by John Hopfield, right? Oh, yeah. Yeah, yeah. Yeah, yeah. Who just won a Nobel Prize in, you know, two or two years ago.

Yes. So John created, um, several-- was responsible for several textbook algorithms, uh, one known as Platt Scaling, another one Sequential Minimal Optimization, which is the textbook algorithm for training SVMs. Uh, even today, it's still, if you use SK Learn, it's there. Mm-hmm. John has discovered and named two asteroids, has a Oscar for technical developments from two thousand and six.

Uh, so if you've ever watched a, a, a Pixar movie, you've seen John's algorithms and work. John has an Erdős-Bacon number of six or three, three and three from either side, and I'm going to skip over, like, twenty years of your career. [laughing] But then jumping into Google at s- you know, working at Google Sciences, you've worked on fusion, quantum computing, climate modeling, and many other topics. Is that more or less right?

That, that's right, yeah. Okay, okay, cool, cool. Did I miss anything important for today or- Um, no. I mean, I've also done, you know, lots of applied math and signal processing and all sorts of fun things like that. Yeah, yeah. I think you also, uh, uh, your, your Wikipedia has a fun story about patents and, um, uh, the iPhone too. The iPod. iPod. The iPod. iPod, yeah, yeah. Yeah, welcome. Uh- Thank you. Yeah.

Thank you for having me. Can you tell us about the ERA is the, I think, the way that the acronym is pronounced? Mm-hmm. Um, and, and I know that there's a lot of different semi-related stuff, um, out there both within and outside of Google. So what can you tell us a little bit about the details of ERA and, and w-what makes it special?

Well, we've been doing, uh, sort of AI for science in Google Research for, uh, more than ten years now. Yeah. And, uh, uh, around ten years ago, it was very much, um, using, I don't know what you call it now, maybe classical, uh, classical machine learning models, you know, things like, you know, convolutional nets or whatever to-- And they were specific models to build to, um, you know, solve specific, um, science problems.

But about two years ago, uh, we got very excited about these more general, uh, LMs that have popped up in the last few years, and we were wondering what can be done with them. And of course, a lot of people have been, uh, playing and trying to figure out what the right, what the right thing to do is.

And, uh, we kind of, um, stumbled into this, uh, mapping, uh, in that we found that many different scientific problems, uh, can be mapped into something we call scorable tasks. So you can often phrase a scientific problem as, "Gosh, I really would like to have a piece of code that, you know, maximizes some score."

And it's surprising the number of, um, different sort of scientific problems you can make a lot of progress on, uh, by mapping into that framework. Well, one thing is, uh, a lot of scientists spend a lot of time sort of building models. They might be statistical models, or they might be, you know, physically based models.

Um, and if it's a statistical model, like in machine learning, your scoring function is, "Well, I have some data set, and I'd like to have the fit, the model, and the data set go up." Uh, and we can talk about overfitting in a minute, but, um, um, uh- That was one of our questions. [laughing] And then that's actually very, that's actually very interesting.

So machine learning is kind of a subset of this sort of scorable task, right? But you could do other kind of things like, uh, especially Michael Brenner, who's, um, uh, uh, the lead author on the ERA paper. He's very, very skilled 'cause he, he, he likes to, to sort of knock out a scientific paper in an evening, uh, now with the tool.

Um, so, um, there's something in applied math called asymptotic expansions and, uh, which is you, you're asking how does this, how does an ordinary differential or partial differential equation, but say ordinary differential equation behave, uh, there's some parameter that has an epsilon in it, and you're trying to say, how does it behave as epsilon goes to zero?

And you could turn-- it turns out you can turn that into a, a empirical task by es-essentially asking that it, uh, it, it proposes some solutions that are asymptotically correct, and you check to see if the, uh, asymptotic solution is correct for like epsilon equals one E minus four or something. And then you, you check that fit, but then, then you ask, uh, uh, you ask Gemini, which is the core AI underneath it, to, to do the mathematical reasoning, try to solve the problem whilst, while also maximizing the fit to the, to the data.

So you can-- Actually, there's a lot of sort of tricks you can do because it's not that the underlying A thing that's, that's altering the code or the underlying thing that's sort of making the decisions is not a random process. It's an AI itself that is smart and knows about things and knows a lot about the world.

You can get a lot of in-- solve a lot of interesting problems because that sort of core inner loop is an AI that has huge amounts of, um, prior knowledge. So that's sort of the, the trick. So we, we, we've been, uh, running around trying to map lots of scientific problems into scorable tasks and trying to solve them, and it's, it's really been kind of fun, and I'm happy to talk about the ones that I've been involved in at least.

Yeah. I would love to hear about some of the more-- So the sta-statistical one is what everyone listening will probably know about. What you just mentioned makes sense. What are some of the other interesting ones? Uh, let's see. That, that are not statistical. Let's see, 'cause we have, uh, interesting ones like, uh, one that we just, uh, put a paper up on archive is, um... Or actually, I think it might be on GitHub.

It's, um, you often run into this in remote sensing because there's always a trade-off. There's satellites flying above the Earth, and there's a trade-off between how, uh, frequently they can revisit a spot on the Earth, what their spatial resolution is, how big the pixels are, and their spectral resolution, so how many bands they have.

And i-ideally, you'd like to have, uh, monitoring of the Earth that's constant and, you know, a, a frame every five minutes at, at hyperspectral resolution at, at whatever, ten centimeters. You can't get that. But, uh, for, for example, to monitor, uh, CO₂, the, the a-atmospheric concentration of CO₂, you can take data from one satellite, uh, that's, for example, that's OCO-2 or OCO-3.

Uh, OCO-3 is actually attached to, um, uh, the International Space Station. But so it, it gets you like a little, uh, strip of CO₂ measurements that are highly accurate- Mm-hmm. -and pretty high resolution. Uh, you can actually try to do, uh, there-- You could from-- Because a lot of it is in the infrared, uh, weather satellites like GOES has some infrared bands, and it, and it takes a picture every essentially five minutes, but the pixels are very large, and it doesn't have such great spectral resolution in terms of it wasn't designed to find CO₂.

So you just ask one to estimate the other, and you shovel other data in, like what's the current weather? Uh, what i- what's sort of the long-term, um, you know, albedo? Uh, and, and so you-- so ERA came up with this very nice, um, this very nice model that can sort of, can do almost like super-resolution, an informed super-resolution of, of, uh, uh, one satellite to another.

So that's like one example. Yeah, yeah. Okay. So a-any, any scientific problem that you can map into this framework. So the, the input is to, to ERA is sort of this mapping, and the output is code. Is that- Well, sort of. I mean, the input is you-- The way we've got it set up in, in the, the, uh, product is you just start talking, right?

And so because a l- a lot of times it's non-obvious to how to do this mapping, although, you know, experts like Michael Brenner know how to do it. So we actually, um, wrote an agent that actually helps you. It sort of talks to you to try to help you define what your scorable task should be. So there's al-- So already there's sort of a instance of Gemini sitting there trying to help you write, uh, a code.

So that's actually sort of an almost like an intermediate result. You start talking to it about your problem, and it tries to produce essentially a, a, a, a Python notebook underneath that, that has a, that has a sco-- essentially a function with a, with a scorable, uh, which essentially produces a score.

And then it starts to mutate that notebook, uh, in a clever way because, again, it's Gemini, and it will try to sort of, uh, keep proposing code that tries to maximize the, um, uh, the score. So what is different about this than just a generally a, a general agentic system that can sort of optimize notebooks?

Uh, right now it's a, uh, it's essentially its own in modern 2026 parlance, we actually worked on this in '24 and '25, but in the modern, modern parlance, it's kind of a specialized harness that, uh, runs an algorithm, uh, which in it for ERA was Monte Carlo tree search. So essentially it's keeping, um, hundreds or thousands of possible instances of notebooks, and then it selects one, and I can explain how it selects one, and it decides, "Well, how can I-- What, what can I do?"

Gemini asks itself, "What can I do to make that, uh, notebook be better?" And then it will make a new one and test it and then put it back into the candidate pool. So you can imagine the candidate pool is actually tree structured because it's, um, uh, you know, every candidate possibly has some children. And what you do is you pick based on something called the, uh, uh...

It's actually, um, um, a fairly standard algorithm in, uh, from reinforcement learning called Upper Confidence Bound, UCB. So you essentially pick-- It's an optimistic algorithm, so it tries to estimate, say, what's the, what's the ninety-fifth percentile outcome of mutation? It tries to estimate that, and it picks the one with the highest bound, the highest optimistic bound. So in other words, it doesn't always pick the best-performing notebook.

It tries to predict, like, what's the current performance plus two sigma of, of, of its guess. And so it, it's always trying-- So it hunts around. So it's like high recall, basically. High recall. It's trying to make its bet to-- so that it, it most efficiently tries to make progress- Yeah. -which isn't always greedily doing the best candidate. Sometimes it's the fifth best. Yeah.

Um, we're also play-- We've also played around where, where it kind of recombines. It sort of takes, uh, ideas from two candidates and, and smashes them together and, um, uh, tries to make a third candidate out of that. How does it seed the initial candidate pool? Well, that's the amazing thing is that underneath, Gemini is actually good at writing code.

I mean, you just ask it, "Write me a thing," 'cause it-- 'cause you have a, you have a textual description of the problem. It isn't just, "Oh, here's your scoring function start." You say a textual description of the function, and you might give it... In fact, we have under some things, like, "Here are five papers that People tried to solve this problem with, and it's kind of smart.

It actually goes and reads the papers, and will actually take a first stab at code. It might not be great or it might, you know, sometimes it has bugs, and it returns essentially minus infinity, but, um, it will then, uh, try to mutate the code and say, "Oh," it'll try to make it be better. So- Yeah...

it's, it's pretty cool. You don't actually have to give it... I mean, you can if you want, give it a, uh, some co- starter code, but you don't have to. How many agents are you spinning up? I guess maybe not agents, or how many different, um, you know, tree branches are you spinning up at each iteration? Oh, at, at every iteration? Yeah. Well, there's a trade-off.

You'd like to do a lot of parallel work, but if you do too much parallel work, you can't learn from previous things. Mm-hmm. So right now we use about, um, uh, the default is, is ten parallel, so you, so you, you try to grow ten leaves at a time. Okay. Um, that seems to be a- about the right, um, trade-off.

When you say you can't learn from previous iterations, that means that the, uh, orchestrator, or is there some sort of, yeah, what, like what's the... That you said that there is some step which is able to like pre-combine or make decisions beyond just like the score. I mean, so yeah, I guess maybe one of the obvious questions is, as a human, when you are doing some sort of ML project, you don't just look at like, oh, there's this one metric that we're trying to optimize.

Oftentimes, there's like orthogonal metrics, uh, sometimes even insights such as like just watching, you know, training curves can sometimes give you intuition about what's going on, or like looking into specific examples. Does it do any sort of introspection like this? Is there, uh- Well, it has, it has the history of...

But what I meant why you can't, um, uh, do too many things in parallel is, is if you, if you have ten parallel, uh, searches at once, then one, the, the number one can't actually see what numbers two through ten are doing. Yeah. So if you do 1,000 at once- Oh... then you're using a huge amount of computation without a lot of cross-learning. Whereas once you finish a little batch, you get the history.

It, it, it sort of, obviously you have to prune it so it doesn't blow up the context, but, but you get the history of, of what it was thinking about as it was, uh, kind of writing the code and the results of the code. So, so it- Mm-hmm... it can learn from its previous attempts. Okay. And does it learn across? Oh, yes. Across, yeah. Yes. Essentially, it's essentially it's, it's like one essentially shared context, yes. Yeah.

So it is, it is- Mm-hmm... sort of thinking- Oh, okay... as it goes along. Okay. Oh, it is- It's not, it's not like it's 1,000 different- Mm-hmm... completely independent- Yeah... branches. You're really pushing Gemini's like long context abilities there [chuckles]. That's right. And you have to do, you have to do the right management and stuff. Yeah, yeah, yeah. Okay. Oh, that's, that's cool.

I mean, going back to RJ's question, um, the like key point here being the, the first, the first key goal is to, I guess, identify what specific score that you are trying to optimize- Mm-hmm... right? Sometimes that is, I agree, like kind of the hardest part of the problem, and so I find it interesting that I'm not sure I'd always trust my agent to do that part.

That part seems like the more human task, um, in the loop. Yeah. It is, and often you have to be careful. In fact, a lot of what you do, it's kind of, it's very meta. I guess everything- Mm-hmm... everything we do is very sort of high level. Yeah. You have to make sure that the, there's no... One common thing is you come up with a scoring function or the agent does, or you do it together, and then the iteration finds a, a, a way to cheat or hole. Like- [laughs]... "Oh, no, I didn't mean that." [laughs] Yeah.

Uh, and so you have to go through and often sort of play and have a loop around it where you kind of iterate like, "No, no, no. I didn't mean that." Or you have to tell it in its instructions, "Okay, well, you know, don't, don't do this." So yeah, there's often iterations, and you, so you, you, you, e- even with agentic help, you don't necessarily get the, the right scoring function from day one.

And in fact, it, it's really neat because, I mean, in the old days, i.e., 2024 or something, you know- [laughs]... a lot of, a lot of grad students would spend a lot of time like, uh, doing scientific software. Mm-hmm. And it's just so much effort to w- write code at all that you kind of try maybe a few things or a few things- Mm-hmm...

that are very related, and then you sort of stop because you have to write your paper, or you have to do your next experiment. Mm-hmm. This thing is kind of underneath kind of relentless 'cause it keeps trying and keeps trying, keeps trying. And so the people who use it are now spending all their time almost at the right level, almost at the scientific creativity level. Mm-hmm. What does it mean to, uh, have a cost function? You know what I mean?

And so that's almost like the essence of the scientific problem. You're not so much now in the details of, oh, oh, I have to import this CSV file, or I have to get this database to work or whatever. Mm-hmm. It's you're now, uh, sort of thinking almost like deeply philosophically about your actual scientific problem, not down in the grungy goop of, of, of worrying about, you know, databases.

In fact, one, one cool thing the, uh, agent can do is actually suggest data, data sets to you. Like, "Oh, have you thought about maybe pulling in this data set and doing a join?" And so it'll, it'll make suggestions about like, you know- Mm-hmm... data sets you can join with, which is kind of cool. Going back to what you said a second ago, um, in terms of agents love to hack things- Mm-hmm...

and, uh, reward hack, uh, are there-- do you have any fun stories or interesting stories about, you know, where things were comedically run off the rails? Boy, I'm blanking. I know other, other folks, uh, have, have run into it. I don't know if it's, I don't know if I have enough details to sort of say what the, to sort of express the comedy of it. [laughs] Okay. But it, it, it does.

It, you, you, you, you kind of get surprised. Yeah. Yeah. Um, I don't know if I have any really concrete, um, s-sorry, I'm, I'm blanking right now. No, it's fine. Um- Yeah, I always like to think of machine learning as like a, it's, it's sort of like the old genie stories- That's right... before Monkey Paw [laughs]. That's right.

Careful what you wish for because you, you're, you're gonna get it. Yeah. That's right. And, and you have that, and that happens very much with this, so you have to be careful. But on, on the other hand, um, it has some knowledge. The nice thing is that Sort of Gemini knows a lot about many things, sort of more than any one person can do.

So it at least knows, especially if you point papers, point, you know, like here, here are five papers that try to do this in some way. So to some extent, it, it does have that genie feel, but to some extent it also sort of does sane things. This is why... Re-remember the whole idea of sort of, um, evolutionary- Mm-hmm... uh, coding.

It's been around since the '70s. Everyone's loved to do that. Uh, like, "Oh, let's mutate, um, list code or whatever to- Mm-hmm... to do things." But the reason why it just hasn't taken off is that random mutation in code space is pretty much worthless. I mean, every- Yeah... just like, well, just like DNA- Mm-hmm...

it's sort of like, you know, most, most things are harmful. So here it's like, oh, no, no, we can actually find... It sort of knows, sort of underneath it knows interesting gradients to try- Yeah... which is why the thing works. Yeah. That, that the underlying loop itself is an AI. So yes, it can, it can maybe overfit and have funny sort of genie problems like you allude to, but it also has some amount of sanity 'cause it sort of- Yeah...

it knows about the world, and it, it, it has world knowledge in it, so. Yeah. The paper though, you were doing Gemini two point five. Mm-hmm. And I think, you know, Gemini has advanced quite a bit. Mm-hmm. Do you have metrics or have you... You know, this is a tool that you're continuously using, and it sounds like- Improving... you're improving. Yeah.

And I'm wondering, like do you internally, have you seen like almost like a phase transition in how effective this tooling has been? Has it-- How dramatic has the improvement been over the last, like, I guess year or two? Oh, well, I mean, a year or two- Yeah... amazing. Yeah. In other words, e-e-every, even every half version of, of...

I mean, essentially, I think it would have been impossible under Gemini two point o. [laughs] Really? And, and yeah. So- I think so. In other words- So-... it wouldn't, it wouldn't have worked. So you started at two point five, and that was like just the- Oh, no, we, we've been trying to experiment with these things actually for a while. Okay.

And things just weren't working, and then they started to work, and then now they're, they're just amazing. So the progress on Gemini major versions has just been stunningly amazing. Yeah. Yeah, I think this is an experience a lot of people have been having, where things were just seemed im-impossible, [laughs] whatever, are suddenly- Yeah... becoming magically useful, like- Yes... really quickly. Yeah.

And so, so if people are, uh, I even say this to scientists 'cause there's some people like, "Oh, I tried whatever two point o and I didn't like-" Yeah. "Oh yeah, that, that was a long time. That was a year ago." [laughs] "That was like a long-" That was an eternity ago. "That, that, that was an eternity ago," right. Yeah. Yes. Yeah.

And in fact, all the, all the-- We even have one of the preprints where we've sort of combined, uh, uh, Era with anti-gravity, and that- Mm-hmm... you know, the whole, that whole harness of anti-gravity is pretty amazing too. And it's... That, that's the one where you can sort of pull in lots of papers, and it can write- Mm-hmm... lots of code for you- Yeah...

and so, yeah. Is that publicly available or is that- Uh, the, the anti-gravity? The... Yeah, yeah. The harness. Well, anti-gravity is certainly publicly- Oh, sorry, sorry, yeah. Uh, the, the Era plus anti-gravity. Uh, not yet. Okay. Okay, not yet. Okay. I find this area really fascinating 'cause like you said, it's been, there's been some form of code mutation out there since the big, the dawn of computer science, basically.

Um, the canonical problem is sort of the over fitting or multiple hypothesis testing problem, I think, which is maybe a little bit better matched to the problem where you're basically, "My hypothesis is now that this algorithm work. Now my hypothesis is..." And, and so that, um, you run a risk that sort of it has exponentially exploded, right? 'Cause now suddenly I have like these, it's like hyper, hyper parameters that I'm optimizing [laughs].

And so that you have this explosion of state space that you're exploring, and so that it m-seems much easier to, to sort of over fit to a problem. What are your thoughts about that? 'Cause it, on the other hand, empirically my experience, I even tried, um, the, you know, sort of open source version of Era.

Um, I kind of strapped it into Claude, and it's running right now, so I can't tell you how well it's working. [laughs] Okay. I'm curious. But yeah, I'll, I'll let you know. Um, but I'm just curious to know, this is a question that's been in my mind about just general AI for science and what, so what are your experiences with this sort of on the, the, sort of on the ground?

I guess there's two questions sort of embedded in your question, I think, right? 'Cause when you t-talk about multiple hypothesis testing, right, there's predictive models and there's descriptive models, right? Yeah. Uh, I, you know- Yeah... can tell I've been doing machine learning for a long time- Yeah, yeah... statistics. That's actually a really good point though. Do you mind explaining that, expanding on that? Oh.

I'm not sure that's something that everyone would, in our audience would be familiar with, like this- Right... this distinction. Especially in, in modern days- Yeah... I think people are trying to sort of obscure the two- Yeah. If you started with LLMs, I'm not sure that distinction would be meaningful. [laughs] That's right. Yeah.

So a, a predictive model is like, let, let's say you just have a, you have some inputs, and you have some outputs. Mm-hmm. And you just, I just wanna build a piece of code that tries to just have the lowest error rate on some data set. That's just a statistical model, right? A descriptive model is actually what science is trying to get to, which is, okay, it should be able to extrapolate because it has sort of the physics or the actual some description of reality that's captured within it, and then you can use it to extrapolate.

Um, because it's sort of like, you know, yes, Newton thought o-apples and gravity, but gravity isn't actually about apple, right? Yeah. If he just fit, if he'd taken as a machine, you know, the seventeenth century machine learning model, like, "Oh, apples will fall, but how about planets?" You know. [laughs] I don't know. I have no data about planets.

[laughs] So who knows what they do, right? So, so when you say extrapolative... Okay, I realize we're kind of going off on a tangent here- No, no, no. Please, no, no, no... but I am curious. Okay, so when you say extrapolative, so there's different ways I could think about this. One of them is, uh, you said a, a model of physics or a model of the world.

Are you talking about introducing explicit priors that, you know, based upon some human intuition, or maybe in this case, LLM intuition? Or are you talking about this is the, the physics is actually learned by the model or the- Oh, it's more-... the underlying process of the world is under one kind of model.

Well, the distinction between those is a little bit blurry, right? Because when a physicist or scientist comes, they use their intuition, and they... Or about, or, or maybe even more than intuition. Like, essentially, there's maybe a, a solid pile of facts that they know about the world, and then they make sure that whatever model they build is sort of consistent with what's known.

Um- Currently in Era, it is, it sort of it is LLM intuition. Essentially, that's what I was trying to say about having a good gradient underneath that, that especially if you point it at existing papers, it will try to build models that are kind of sane underneath because if it's, again, if you, especially if you give it guidance like, "Oh, be sure to incorporate this and this," or, "Look at these papers to get these things."

So it will-- there is a-- you can introduce a bias towards, um, certain model choices, and it will have a bias because it, it just, its own little sort of world knowledge is accumulated inside of, inside of itself in, in, uh, in sort of pre-training. Can you give us examples of what that might look like? Is it modeling something in a way where it's actually, uh, there's different...

For example, if you're doing something with partial differential equations, there's these, you know, formalisms people have like, uh, neural operators, for example, or where you can embed a, you can kind of encode a differential equation in some sense or, uh, I think that's called physics-informed neural networks or something.

Mm-hmm. Is like one for fluid and kind of PDE-type modeling systems or for, let's say, molecular systems, there's oftentimes this idea about equivariance. I mean, are the models like picking up on these like, you know, tricks which have been developed in the literature, or are they adding some weights for some physical prior or something?

Uh, yeah, kind of what does that look like- Oh. -when it, when it introduces like a physics-- when it introduces some sort of, um, you know, knowledge like that way? I don't know if I have enough data to sort of say, "Oh, you know, seventy-three percent of the time it does this." Okay. Sure. But, um, especially when you point it at existing papers, it will try to, in fact, it will do very well at adapting the, the, the methods that are described in the papers, uh, for the problem.

Um, in fact, it will do amazing job. You can actually often just recreate or reverse engineer a paper. That, that's again what Michael Brenner actually likes to do this. Uh, he'll, he'll say, "Oh, that sounds like an interesting paper." Mm-hmm. Uh, we did this actually for the, uh, we had this thing. It was actually kind of a hack.

I, uh, suggested this to Michael, where, uh, there's this one MIT professor who, um, he came up with a, a, a, some code to do, um, essentially if, if you have a rooftop, uh, if you have a rooftop with a fixed, uh, area and you want to sort of maximize the amount of solar, uh, power you capture over a day, sort of solar energy, uh, you can build up, which of course captures more sunlight, and you can sort of build, you can have it design a widget, you know, with involving mirrors or struts or solar panels, uh, just sort of stock at whatever, uh, uh, you know, angles or, or sizes you like, and, uh, usually with, with a, with a maximum height, and then try to figure, let it sort of explore that design space.

And I believe, Michael, we can ask him, I believe it actually just-- I don't think he actually installed the simulator. I think, I think the code just, it just reproduced the code, uh, 'cause it has like a coding agent inside of it. Yeah. So it just reproduced the code- Got it. -from the paper and sort of figured it all out. Yeah.

So y-yes, it's, it's very good, especially if given a pointer to what other people have done. It's very good at kind of like, oh, I haven't seen it try equivariant modeling. That, that can get very hairy if you know about, uh, Clebsch-Gordan coefficients. It's, it's pretty nerdy. They're fun, yeah. [chuckles] Yes.

So I don't know if it'll do the true equivariant stuff, but it, it actually knows a lot. Um, I remember actually when, uh, Gemini two point five came out, I know this is not exactly about Era, but I, I remember sort of thinking, "Oh, this is a new world," when like the day two point five came out because I said, "Hey, uh, Gemini two point five, can you write me, uh, some boost decision tree code?"

Mm-hmm. And it did. [chuckles] And it worked. It just did, yes. [chuckles] And I said, you know, "Yeah, this is the, this is a new world." [chuckles] It worked. Uh, so y-yes, I think it's, to loop back to your question, I think, yes, if you give it sort of guidance about, oh, you know, it's important to put this kind of thing in, it will.

And so it, it, it won't necessarily, at least not that we've seen, discover completely new, uh, like if you didn't know about Clebsch-Gordan coefficients, I don't know, I mean, you know, you didn't know about something, it won't completely discover a new kind of physical model from scratch, but it will certainly, if you tell it about interesting constraints about the world that are known, it will certainly follow them.

I don't know if I answered your question. So, so there's still room for humans for the next year or two. Oh, in fact, there's, uh, going back, I think there's totally room for humans because I don't know. I mean, we have co-scientist that tries to help you come up with, um, sort of hypothesis generation.

But really, I still haven't seen, um, sort of the creativity and the philosophy and the sort of the, the sort of the careful rigor. You totally need the humans. I, I don't see humans going away. They can make, uh, strange suggestions, and, uh, I've used co-scientist actually for an interesting problem in geochemistry, and I learned about a new kind of ion I didn't realize happened in magma.

But, uh, I-- but and so it'll tell you interesting things, and you'll learn stuff, but I don't think it sort of substitutes for human creativity. You know, going back, we were just talking about two years, two point, you know, two, you had Gemini two zero to two point five, and this was like you already saw a leap, and now it's been another year or two, and now we're at three point five, and, you know, there's, you're saying this is working much better.

I mean, whenever you look at a graph, you know, you can, if something looks like an exponential, it can either, you can either be on a sigmoid, or you can be at the beginning of a takeoff, right? I guess every exponential turns into a sigmoid eventually. Every exponential turns into a sigmoid. Yeah. But the question is like, where are we on that?

I mean, I, I guess I'm a big believer in sort of the whole jagged, um- Jagged frontier. Yeah, the jagged frontier. And so certainly, at least what I see, I mean, I don't know what's going to happen in a couple of years, but yes, there, there's some big spikes out in jaggedness in terms of coding ability and just gathering knowledge and, and finding related things and, and that's huge and wonderful, which I think is great for scientists.

So far, it's kind of less in terms of, uh, rigor and, um- Uh, uh, uh, we can talk about things like the International Math Olympiad, but, and math in general. But, but, but in terms of sort of philosophy and creativity, I think it's still kind of not-- And I, I don't-- Maybe it'll-- Maybe everything will-- People, some people are saying everything's gonna inflate and pass- Mm-hmm.

But I'm still seeing a lot of very strong jaggedness. So I, I can see, well, maybe it'll get to be extra good at coding and extra good at, at fitting models, and extra good at making suggestions and things, but I don't know. So far not. Uh, so far you need the humans. Yeah, I wanna get back to the question about the multiple hypothesis testing. Oh, sorry. Sorry. No, no. We completely derailed. Sorry, yeah, yeah. Totally, totally love the, the tangent.

Um- Multiple hypothesis testing is when you make, have a descriptive model, and you're saying, "This is the way the world works." And you have a bunch of data, and you take a billion darts, and you throw, and so you have to be careful there's, um, something called false discovery rate. Yeah. Mm-hmm. Right?

And so the question is, is this finding descriptive models or is this finding sort of predictive models? And fundamentally, the scientist is there to make sure that whatever is saying is descriptive. We haven't been able to make a system so far, uh, out of these pieces that, that can really sort of discover completely new physics or completely new science, but this is sort of a power tool to help you discover completely new science.

So, so I, I maybe I'm trying to unask your question about multiple- Yeah. But I think this gets at the heart of it, yes. Yes. But then you're saying, what about just pure over-- Okay, let's set aside. It's not, it's not trying to figure out a descriptive model of the world that's still up to the scientist. But what about just plain old overfitting? Yeah.

Yes, you have to be very careful because it's a power tool that can, uh, I shouldn't probably say it should, it can slice your fingers off. No, but you know what I mean? It's like you have to be very careful, and you have to be very rigorous. In fact, now you have to be more careful- Yeah... and more rigorous to not fool yourself.

You really, really need to be just excruciatingly careful about having, you know, very hidden, uh, holdout sets you don't look at. You, you have to be just super, super rigorous to make sure that you don't completely-- 'cause it is a total power tool. So the question to how do I not slice my fingers off is you need to use the same techniques but be very careful with them. Yes. That's a very clear answer that [laughing] I get for free. Thank you. Yeah.

I've, yeah, I actually don't think I've heard any guests say that. Yeah, it's like it is, I think, a very important skill, maybe one of the most important skills in the new, um... People talk a lot about taste. Yeah. But, and maybe this is a variation of taste, but like- Yeah, the, the taste is like the other side, right? It's like the rigor. Yeah. It's like the... Yes. Uh, yes.

In fact, if anything- Yeah, taste or rigor, which one's more important? Well, I don't know. I think people, at least the way I am viewing it is, is s-- I mean, the aspect of that research, researchers are software developers, which there's a lot of overlap in a lot of fields. I'm seeing that software engineers are-- It's almost like obviously there's a lot of, um, concern like, "Oh, no, what am I gonna do? Uh, you know, coding seems to be, you know, getting, uh, automatic."

So I think, I think there's sort of both. I think there's a lot of people get pulled into, "Well, I'll, I'll be the creative source, so I'll, I'll try to figure out new science. I'll try to figure out new products. I'll try to sort of really, re- be very, very creative." And I again, I'm a strong believer that I don't think that's gonna go away.

There's also people sort of pull towards rigor, like, "Oh, I wanna make sure this doesn't, that this doesn't crash. I wanna make sure this scales. I wanna make sure this isn't wrong." I think you need both, and I think you need people who, who are really good at both. But they don't necessarily have to be the same people.

But yes, I think you need-- I think this is even broader than science, just as sort of software, uh, engineering, uh, evolves. Yeah, it'll be, you know, the people who will bring the creativity and the people who will bring the rigor, and I think those will be sort of anchors. Some other things that I've seen are related work out there.

There's a, a really cool leaderboard for, you know, I call a leaderboard, agent leaderboard for scientific problems from Stanford. I don't know if you're familiar with it. It seems like a really interesting idea to me to have, you know, sort of different agents kinda competing on the leader. So it seems like if you squint a little bit, what Era is doing is, is kind of a leaderboard, but it's internal, and it's recombining ideas, whereas what are your thoughts about this?

And do you s-- Is that like a thing that you guys are working on, and is there problems with that or advantages to that? Ironically, you know, the whole Era project actually started because, uh, uh, people may not realize Kaggle is actually part of Google. Oh, yeah. Uh, and so it was called the Auto Kaggle problem.

So it was actually like, that's what it was. Let's try to- Okay. Yeah... let's try to- Right... um, um, um, have a system that can sort of, you know, win at Kaggle competitions. So that's sort of why- Mm-hmm... it sort of has this shape. That's sort of how the project started. And, um, it goes back to sort of overfitting, right? If you've ever actually competed in a Kaggle competition.

I have done Kaggle competitions and, or I've done one. It, it, it is a really interesting phenomenon because there's this overfitting is like rampant. Rampant. Yeah, yeah. And it's really impressive how people can overfit to certain datasets- That's right... in a way that is, yeah. Or even we had a, we have a fun project I can talk about- Oh...

over if you like, um, that tries to mitigate contrails- Mm-hmm... jet contrails. Yeah, yeah. I'm happy to talk about that if you want. Uh, and we had a j- a contrail, uh, Kaggle competition, and people actually beat us, but they found that we had a, a half-pixel, uh, error in our labels. [laughing] And they, and they-- 'cause it had to do with the center versus the lower left. Like, what, where is zero zero? Is it in the lower left of the, of the pixel, or is it in the center of the...

You know what I mean? Yeah. So they found that and exploited that and, and squeezed whatever, a little bit extra stuff, uh, 'cause it turns out when you, uh, make artificial data and you rotate it- Yeah. Mm-hmm... you, you have to make sure that That you take into account that half pixel offset. Yeah. [laughs] So yes, people or people themselves will be-- act like these LMS- Yeah...

uh, and, and try to sort of reward hack on these things. So it sort of b-goes back to, um, what is it? Goodhart's law? You know Goodhart's law? Yeah, Goodhart's law. Yeah, yeah. Any, any- Maybe we can quote it, though. Yeah... uh, let's see. Let's say any metric that becomes a target h-- is no longer good as a metric. Yeah. Uh, and, uh, so that's the...

I mean, it's good, a-and it's just you have to be, you have to be very, very careful and very-- you have to, again, you have to have like layers of rigor. Like, okay, but we'll do this, and we'll optimize for this, but you have to realize, okay, that's just now Goodhart's law applies, and you have to be careful.

And so that's a lot of reasons why the whole AI field has been kind of constantly exhausting these, these things because, again, Goodhart's law applies individually to every, to every leaderboard you make. So again, it's sort of you just have to step back and be very, very careful. I-I- Yeah... maybe that's not- No, no, no, no. That, no, that, that's really useful to my thinking.

This is, you know, w-as we've had guests on, it's been a recurrent theme of how do you manage all this, the complexity that's introduced by LLMs and agentic science. I think my follow-up follow-up question was about overfitting in Kaggle. Mm-hmm. Um, yeah, it is if you, you had an auto-auto Kaggle problem, and then the question is, given auto Kaggle, how often was it successful?

I mean, assuming you probably just ran this on, like, all of your Kaggle competitions or something. Well, we tried it on various, um, like what they call playground competitions, and it did very, very well on the playground competitions. Um, uh, we've entered into different competitions, some of them.

It turns out, uh, there's, uh, in the last few years, just the, the number of leaderboards and competitions and whatnot- Mm-hmm... have just exploded far beyond Kaggle. So, uh, we've done very well in some of them. Like, uh, one thing we're super proud of is the whole, um, CDC set up this, uh, competition where you try to predict next week's the number of, uh, COVID and flu cases that will happen in every state and, um, uh, territory in the US, uh, and you try to predict a week in advance.

And Era did super well on that. It's funny because in some sense, Google invented the concept of using data to track, uh, disease progression with Google Flu. Google Flu. Uh, it's, so it's kind of funny that you're sort of going full circle twenty years later or something like that. That's right. So that did very well.

Other ones where we've entered, um, uh, we weren't quite as good often because people are very... Again, you know, you have to someti-- uh, sometimes how well you do in these competitions is a, um, measure of how much sort of TLC you put into it and how much you're willing to squeeze- Mm-hmm... the last point zero zero one. Yeah.

And so the, so it was, I mean, Era did well. It, it got you pretty close, but you, but we didn't, we didn't close the jump in the last- Mm-hmm... whatever, thirty places or whatever because no one was there to shave the last, you know, point zero zero one- Mm-hmm... off the thing. So. Yeah, yeah. Is it very iterative like you get to it, you know, it sort of-- I saw the charts in the paper and, you know, you sort of get these step changes as it discovers something and then, and then flat, and then...

So is it very much human in a loop like, "Okay, you've stalled on the problem. Like try this," kind of thing? Yes. Okay. Yes. Uh, at, at the outer loop, which is I think that's the almost like the more fun creative part. Uh, so yes. "Oh, here, have... Look at this paper. Oh, you're doing something bad," or you know what I mean? So- Yeah... it's almost like having a, a hyper-eager, uh, grad student or something who doesn't sleep.

Uh- [laughs]... uh, and you sort of tell it things, and you, and you- Yeah... sort of guide it around. How often does someone intervene versus like what is the outer loop look like actually? Uh, it might run for a few hours and, and come back and give you, uh, some examples, and then you would, um, you know, you can, as, as far as you like- Yeah... you can, you can sort of keep trying and keep- Yeah...

poking at it. And- So that's is very much designed to be human in the loop then? Yes, yes. Interesting, 'cause a lot of the other tools that I've tried tend to be very the one shot. Well, I guess it depends on your definition, right? Yeah. I mean, it's obviously you talk to it, and you start it- Yeah, yeah... and it will go for some number of hours and come back.

And then, but of course, then you say... But then that's where the human creativity kicks in, and then you're sort of doing the outer loop where you sort of every, you know, depends if you want to sleep, but, you know, every few hours you, you go, and you give it a- Yeah... another try, and you- What kind of budget are you giving this thing? [laughs] Like you, you, you blew through a million dollars accidentally kind of thing.

I don't actually know because we're using sort of, you know, internal calls to- Yeah... uh, to, to Gemini. So actually I, I don't- Oh... actually know. So but okay, so there's token budget, but then there's also like I'm solving a problem that is computationally expensive. Oh, yes. That also it, um, essentially underneath it 'cause, 'cause the, the scoring function itself might have, you know, Monte Carlo estimation or, or whatever. Yes.

So you actually end up, you can actually end up, uh, using a lot of compute, uh, uh, to just, just even do or simulation. Like if you have a simulator inside, it has to run a simulation. Uh, so yeah, you can, you can, you can spend a fair amount of just- Yeah... CPU or GPU. So my little experiment with Era, um, Era and Cod is, is to build a, a neural network for some classification problems. Mm-hmm.

And, and so they obviously, like if you have enough data, then, you know, larger networks work better, but they're more expensive to train. And you get to, you start to run into a question of how do I manage my budget if I have a fixed budget so that I'm spending my, my, uh, my, my dollars on the most ef-effective solutions. That's right.

And I think that's still something we need to figure out. But it's of course, it's no different than if you have a grad student and they're trying to s-s train a very, very large, uh, neural network or a very, very large data set. Yeah. They themselves have to... There's something like, "Oh, is there a scaling law? Can they extrapolate?"

So it's not-- I guess it, you, it's an, the same problem, but maybe more urgent because it, it just, it runs into this problem because it's so relentless. It runs into the problem much quicker- Yeah... than a grad student could. One of the things that you optimized was- Um, contrails. Um, can you talk a little bit about that? Uh, well, uh, let me maybe spend a minute or two talking about the contrails problem.

Yeah, yeah. Uh- For context, contrails, not chemtrails, which is a conspiracy theory. [laughs] Uh, yes. Uh, although you should also dislike contrails, but maybe not for the same reason. [laughs] Okay. So contrails are, uh, if you've ever seen, uh, those white clouds- Mm-hmm... form behind jets, those are called condensation trails or contrails.

And it turns out they add, at least a-according to the estimates that people have, uh, about one percent of all anthropogenic, uh, uh, global warming is caused by contrails. Why is that? I can just talk about the, maybe the physics of that. So it turns out that, uh, there, there's actually two, uh, countervailing effects. Contrails are, are, uh...

Well, sometimes if you've ever seen them, they, they streak in and they kind of go away. Those don't really do anything. But sometimes they last for a long time, and you'll just see in the cl- uh, in the sky just like, almost like a waffle of, of, of, of just, well, persistent contrails, they're called.

And, uh, there's two effects that they have. Those are thin white clouds, so they reflect sunlight, but that only of course happens during the day. It turns out, uh, all, uh, all objects emit something called black body radiation, and the Earth does at whatever the temperature is, about three hundred Kelvin. It, it's in the far infrared, around ten microns.

And at those, uh, wavelengths, uh, contrails have very low albedo. They're almost essentially black. Okay. And so they'll absorb a little bit of the outgoing, uh, infrared radiation and then re-emit it both directions. So essentially they'll reflect some of the outgoing heat, so it'll trap heat like a blanket. Um, and so because that happens twenty-four hours a day, uh, they tend to be, uh, warming.

And it turns out it's a surprising-- Again, there's, uh, some uncertainty about it, but you know, uh, contrail cirrus, cirrus that sort of comes from, uh, from, from contrails, um, might, might cover, especially in places like Europe, which has a lot of air traffic, a few percent of the actual sky is covered by sort of additional, uh, contrails.

Which so it adds, in those places like Europe, it adds about one watt per square meter of, of forcing, uh, locally at least. Uh, which means that, uh, just to give you a sense, all of anthropogenic, uh, uh, warming sort of averaged across the whole globe is about three watts per square meter. So in places of high, uh, airplane traffic, uh, it can be a lot of warming locally. Uh, so what can you do? Well, um, it turns out, uh, c-contrails are caused by areas in the atmosphere that are ice supersaturated.

They're a little bit like rock candy. So like when you have, uh, rock candy, you get a water solution that has too much sugar in it, and any little, uh, you know, little bit of sugar in it will just crystallize all the, the, the, the sugar out. Just like in this contrail, there are these regions, they tend to be kind of pancake shaped, only a few hundred meters tall.

And if you fly through it, the jet exhaust has a little bit of, uh, moisture in it, which, uh, will turn into droplets and then freeze. And then for every gram, if you're in this bad region, for every gram of water, uh, ice or soot you put out, it's about ten kilograms of- Whoa... water gets sucked out.

So there's this enormous ten thousand to one curing ratio. Um, so it's a big problem. So what, what you can do is you can figure out where these regions that are invisible, of course, uh, these regions of ice supersaturated are, and then tell the plane to go underneath, and you only have to drop essentially, uh, what they call two flight levels.

So it, it actually, it costs a little bit of fuel, but not very much to kind of avoid these sort of bad regions. So we built a system that sort of looks at satellite images and tries to detect where contrails are, so we have essentially a continuous monitoring system, and then try to build a model of, of-- because it turns out the weather models are not quite accurate enough to, uh, to find these, uh, places of ice supersaturation.

So we built a custom model, again, like a convolutional net or a UNet or something, uh, to essentially to predict where they're gonna happen so that, uh, then you-- we give, uh, maps to a flight planning software so that they can dodge it and, and inexpensively reduce the, the, um, the climate impact of, of aviation by a lot.

What's the physics behind why you can predict that? Right. Is it just I see it in the satellite, and then tomorrow I think it'll be there because- Oh... planes go to the same place? Or is it- Oh, no, it's because you're trying to detect these, uh, regions of ice supersaturation because they're very, very persistent. Right. Essentially, they're- Oh, they're persistent. Oh, yeah, yeah.

I mean, no one knows exactly, but they could last for days. I see. Essentially, they're caused by, they think, sort of warm, moist air being injected just at the boundary of the tropopause between the, uh, just at the bottom of the stratosphere, and then when humidity g-gets up there, it sort of sticks there for a long time and then gradually dissipates.

Got it. So, so it's, there's just a sort of- They're like bad spots in the atmosphere you don't want to fly through. Right. Okay. And so once you've established that, it's probably good for a couple of days at least. Uh, well, you have to keep predicting- Yeah... where that is. Yeah. And the, the models you're using are, you've mentioned like CNNs or something like that. That's right.

And we haven't replaced those with era level, uh, models yet, but there was a very interesting problem that came up, which is you sort of wanna know, well, just how much warming did this contrail make, and how much did it add to global warming? Uh, because, for example, you might wanna find the biggest ones because there's some fuel cost and maybe it costs a bit of money for the, uh, air-airplanes to avoid it.

So you say, "Well, gee, I'd like to kinda know how much it did." But that's actually, uh, what they call a counterfactual problem. Like, okay, you made a contrail and certain amount of infrared radiation happened, so we can measure that. Uh, if you're careful. But would have happened if there hadn't been a contrail there. Yeah.

That's a very difficult thing to estimate 'cause you can't access the universe where, uh- Wait, that didn't happen [chuckles]... that didn't happen. So you have to make these things called counterfactual models, and those, those are ac-- uh, if, uh, I don't know if your listeners know, counterfactual models are actually pretty tricky to, to fit and make.

Uh, and we were-- there was-- remember there were two, there were two, um, the reflecting the, the, the sunlight, and then there's the infrared. It turns out the ref-- uh, measuring what the, uh, effect of reflecting sunlight is actually more difficult, and we were actually stuck on it for two years.

We, we had a model that worked okay, a counterfactual model for the outgoing long wave radiation, but not for the reflected sunlight. ERA actually helped us find a, a, a model. It sort of searched all the confounders and sort of figured out like, "Oh, how, how can we estimate it?" 'Cause we had, again, we e-even had like test, test code on, on sort of artificial-- 'cause you can kind of inject artificial, uh, make artificial datasets where they're sort of injected, um, contrails and sort of figure out, "Oh, well, we know how much it was because we, we injected it."

And so a-again, our own attempts didn't even pass our own, um, tests, but ERA's thing actually did and, and, and sort of unstuck this problem. So yeah, we're in the middle of writing up a paper. Uh, we have a paper about the outgoing long wave radiation, but we have a paper, um, that, uh, it's not submitted yet, but we've talked about it a-at, uh, at EGU, I think, um, where, um, we actually solved this problem, so.

And, and the, the models that ERA comes up with, uh, are they just like a big monstrosity of, of, of, of code, or are they like pretty basic and it's just you needed the intuition to develop that? Yes. It's actually, in this particular case, it was actually more of the latter, that it essentially sort of helped identify what the-- It's, was a very simple model with some- Hmm...

just a s-- uh, uh, some number of confounders that we just hadn't, we just hadn't tried that combination before, and it worked very, very well. So yeah, it actually sort of came up with the, the, uh... And it was sane in, in retrospect, so that was, I think, a, a big win. Yeah. That's interesting. I know you've done a lot of work in climate. What other, other, um, stuff have you done?

I think I talked about this, right? The, the, I talked about the CO2 thing. That was pretty fun because it's this-- It's still quite a, a... The reason why estimating CO2 in the atmosphere is an interesting problem is we actually don't know what the carbon flux is into the bio-- in and out of the biosphere. I mean, we do.

We know that the biosphere captures, right? We emit a bunch of CO2 out into the atmosphere, and some of it gets absorbed into the ocean with sort of mostly inorganic chemistry, some, some phytoplankton, and a lot of it gets absorbed on land. Uh, but we, but the error ba-bars about what happens is, are moderately large, and the error bars fifty years from now are very large.

Like the models in 2100, we don't know how mu-- how the biosphere will react to the ever-increasing temperatures and CO2, so we don't actually know how much the CO2 will absorb. And the, uh, the, the, the error bars are 300 ppm of CO2 just from the uncertainty of how, what gets absorbed. And just to point out, you know, right now there's about, what, four forty, four fifty ppm. So it's huge.

I mean, it could be seriously amazingly awful or not great, but you know, it, it, the, the 300 ppm is like enormous, uh, uncertainty, so it'd be really nice to figure out, you know, can we, can we reduce that? So this is like the CO2, uh, concentration is like one step towards that, um, solving that.

And I know that Google has o-- made some really big, um, improvements in climate modeling and ge-- and weather prediction as well, right? Is it-- I was at NeurIPS this year, this last NeurIPS. Yeah. And, and, um, and I stopped by the climate track and, you know, I maybe only had a chance to listen to two talks or something, but I was-- it really blew my mind the sort of step change, I think, that's happened in the past, I don't know what it is, maybe ten years or whatever, in terms of climate modeling, and I know a lot of that happened at Google.

Can you talk a little bit about what has happened in Google and o-other places that has made that, allowed that really big transition in, in climate and weather modeling? Uh, okay. So let me... People often sort of collapse climate and weather together. Yeah. Right. Well, because they're fundamentally the same physics. Yeah.

Although, uh, at least a-for the atmospheric physics, they're, uh, obviously when you start having ice and land, you know, c-climate is long-term weather. And so, uh, the complexity of a full earth system model, which is a climate model, is much, much bigger than an atmospheric model. Like, you have to actually measure, uh, what's the water flux and the CO2 flux in and out of the land or what will happen with ice. And so there has been a step change with weather models.

Sorry, sorry, I wanna make this- Yeah... so weather is up to fifteen days- Okay... approximately. Yeah. Because, you know, weather itself or the atmosphere appears to be chaotic. Uh, uh, I'm, I'm hoping, I don't know if I should explain chaos. [chuckles] Uh, essentially it's the butterfly effect, right?

That small perturbations, like a butterfly flaps its wings and the weather will be completely different in, you know, two or three weeks. So weather is trying to predict the actual, um, trajectory of the atmosphere over, say, two weeks. And that's now, that has been a huge step change, and that's because that's been a lot of not even the new LLM stuff.

That was based on, you know, the 2018 era, um, machine learning stuff and just a large amount of data and a large amount of compute. So it's been, there's been a lot of sort of very clever work and a lot of it from Google making new, um, new weather models, and it's been, it's been great. And in fact, uh, we had a, a, a, a really neat breakthrough 'cause now we can a-apparently predict Tracks of cyclones, tropical cyclones, uh, much more accurately many days, uh, in advance.

And so places like, um, Jamaica had-- got hammered by a terrible, uh, hurricane, and, uh, a lot of the classic models didn't actually predict it, partially because it's often that the in-- especially the intensification, it's all being driven by the, um, uh, what the surface temperature is. 'Cause hurricanes are, people might not realize, are heat engines. Essentially, they, they convert sort of heat in the ocean to big atmospheric, um, um, motions. So weather's been great.

Climate is much more difficult because you don't actually care about-- You're not trying to predict whether it's gonna rain in Seattle in twenty seventy. You're trying to get kind of like averages. And what makes it difficult is that it's what they call non-stationary, so that in fact literally the ph-- the, the, it's like the underlying physics or the underlying, like, you know, plants are behaving differently and, and ice behaves differently.

And, and, and so it's a, it's very, very difficult to use sort of classical ML on sort of true climate models. And so that's why sort of the whole discussion you guys had about, you know, what we're talking about, about descriptive models and multiple hypothesis testing, that is incredibly severe in climate because we don't-- we have no data from thirty years from now, and we don't wanna wait thirty or fifty years to find out whether we were right or that we overfit.

So whatever things we do, you have to be kind of careful and try to peel off sub problems and, and the problem of, of exactly how do you inject, how do you build a big model that can predict into the future, uh, but is still constrained by what we know. It's a fascinating problem. Uh, I think it's still unsolved, but it's a, it's a great problem to have because of, again, these uncertainties.

We really would like to know what will happen in, in sixty years to the climate. So it's still, it's a thing, it's a very, very interesting problem to work on. But so far, AI has not revolutionized it 'cause it's very, very resistant, again, 'cause of this data problem. It's, it's a low data problem.

Does the butterfly effect, the chaotic, um, nature of weather, does that also impact climate? Or is the timescale so large that you have a closed system for which, you know, maybe it's oscillating between poles or whatever, but it's, it's sort of when you look at it at that timescale, it's more stationary.

It's unfortunately non-stationary in a different way. Oh. But the, the original sort of whole, uh, chaos thing was, well, uh, maybe many people came up with it, but, uh, I mean, meteorology, it was back to a person named Lorenz, uh, who had this sort of very model, uh, very simple model ODE. So the difference between climate and weather is, is weather is where are you on the attractor, and climate is about the statistics itself of the attractor. Right.

The problem with, with climate is that we're altering it, so the attractor itself is changing, is moving. And there could be, and there could be-- Everyone talks about tipping points. That means that the attractor suddenly changes, and the trouble is that's very, very, very difficult to predict because- So even the attractor's shape changes quickly. Or could. Could.

And the trouble is when you run a simulator, you don't know, like, is it, did it go unstable because my model's not great, or is it an actual physical instability? Interesting, yeah. And it's extremely difficult to tell the differ- the diff- So what do you do? Especially when, I mean, to me, it strikes me that not only do you have, not have future data, you really don't have much past data.

You can do- Correct... some measurements and ice cores and lots of stuff to try to do that, but there was nobody with an instrument a hundred years ago measuring things. That, that's right. So if you have annual data or whatever, maybe you have, if you're lucky, fifty data points in any one location. Right. So whatever we do has to be very constrained by, uh, what we know.

But it's just very diff-- I'm just telling you sort of the horns, the dilemma people are on. So people make these-- In fact, people in applied science in general, I would say climate is the most extreme, make these things called process models, where what you do, and I've seen the code, uh, "Oh, well, you know, I'm gonna be reductionist, and I'm gonna sort of take the horrible, complicated climate thing and sort of boil it down to a thousand pieces, and then I'm going to, you know, go find the expert who wrote a paper about, you know, piece number seven sixty-three, and he fit a cubic to some data."

Like it has-- Like, for example, one thing that's very mysterious, which is related to contrails, is how does ice behave in clouds? It turns out you might-- Again, everything is complicated once you dig into it. [laughs] But it turns out that, uh, like when you make a contrail, how long does it last?

Well, it depends on-- Because the way, uh, contrails can evaporate is ice starts to f-- to accumulate, as I said, and then the ice crystals get big, and then they fall. But of course, how quickly they fall depends on their shape, which is not known. And how much does the, the, the contrail mix from the, from the moist inside the contrail out?

Again, people have approximations, but they don't know. And so the, uh, the uncertainty is very much confound. And it's not just, "Oh, uh, John, who cares about contrails?" It turns out that the actual physics of, of microphysics of ice has very strong implications about what climate models do, and it's sort of we just don't know.

So I'm trying to say it's very gnarly, and it's not a solved problem. My hope is that with tools, maybe not like today's era, but maybe tomorrow's era, because it-- Remember, you can, as I was, uh, saying before, it not only can fit data, it can read papers, right? Yeah. And the question is, it can read a lot more papers than we can.

So maybe, maybe we can integrate all the data that or all the knowledge that people have carefully evaluated, much more than any one person writing a piece of code, and fit data. I mean, that would be, that would be utterly glorious. Uh, we don't have that today, but, uh, that, that's sort of one of the hopes that I have e-even for a, a more amazing tool, uh, in the future, is something that really can write code In a sane way, m- even much more sane because it'll be constrained by all the scientific knowledge that we've accumulated so far.

Yeah. That would be amazing. We don't have that today. I mean, that strikes me as being very similar to biology. Oh, yes. Oh, boy. [laughing] Right? If you've ever played bi-biology or even looked at biology, there's so many exceptions and so many hacks in the biological systems. Yes. Oh, boy. So yeah, uh, it would be amazing if we could have a thing that could really integrate all known scientific knowledge with data and try to syn-synthesize sort of new models and, and new things.

I think what you're saying is that AI can be an unlock here to some extent because the, the models are so piecemeal, they, uh, ne-necessarily piecemeal, and so that being able to assemble the jigsaw puzzle, not to mix metaphors, but to assemble like a really, uh, this jigsaw puzzle, um, having just scale and capacity actually helps a lot.

That's right. The one things that these AIs have is somehow, you know, humans, even I'm pretty well right, I think, but it's just difficult for me to kind of integrate across the N squared different pa- I've-- My N-- If N is the number of papers I've read in the life, it's pretty big, and it's just difficult for me to even do that N squared thing.

But somehow there's just so much data in those billions and billions of parameters, and you can also give it access to read PDFs, that it can somehow start to pull things together that people wouldn't do it. So that's again, I'm starting to see little indications of that inside of Era. I'm not claiming that's what Era does today, but yeah- Yeah... that, that's sort of my hope of where this is gonna go.

I think I've heard a lot of people suggest something like the, the route to intelligence is to combine LLMs with some form of search. So it's actually amazingly like Era- Mm-hmm... doing that, right? Something which, you know, is a, maybe a very, a very, a very strong database lookup with a, a, a good search algorithm is one way of- And, and of course, I mean, there was the whole...

I mean, people still do, I guess, the whole RAG thing, of course. Mm-hmm. Yeah. Um, and i-if you think about it, Google itself, you know, uh, the 10 blue links things, it was a, or is a, a form of AI- Mm-hmm... even before we had LLMs- Yeah... right? Because it was like you can, you could cast yourself back to whatever, 2010 or 2015, you could ask Google about literally anything, and it will tell you stuff, right?

[chuckles] Yeah. And so it's- Surprisingly well, actually. Yeah, surprisingly well. Yeah, 'cause somebody on the internet has written about it probably. [chuckles] That's right. So if you can match that. In fact, that was one of the reasons why I wanted to come to Google, is just that was such an a- was such an amazing thing, right?

I wonder how much of our audience did a search pre-Google and just know how bad that experience was. [laughing] Yeah, I remember- It was, yeah... 1998, I think, or I think it's when Google-- Like I was using AltaVista. Yeah. And like, I, I don't know, Google had just gotten released, and I used it, and I just... Sorry, sorry, uh, uh, Digital. Yeah, I remember that too.

I just, I just dropped just like a, like a hot potato or something and started immediately using Google. So it's, it's-- So that is sort of a form of AI. Mm-hmm. And so yes, it might be, uh, yeah, i-it could be that just having access to all of that and sort of keeping it in mind- Yeah... at the same time.

That model of sort of scientific discovery, and as much as it pans out, is kind of comforting too because it is reductionist, so that you can look at the individual parts and understand them. So it found the exact things to assemble, but they're all actually maybe fundamentally things that people have invented or, or it's done iterations on and, and so tho- all those little pieces are individually understandable, and then you can also put them together into a coherent picture.

I think for a lot of problems like biology or climate science, I don't think we would trust the answer unless it was in that shape. Yeah. Because if there was some giant black box model that said, "Oh, this is how a cell works," it's like- [chuckles]... do I believe it? I mean, I don't know if I believe it because I can't examine it, so.

But I mean, to, to argue against that though, like if it, if it works really well. But you'd have to gather... You'd have to... I mean, you have to test it obviously. But a, but a statistical model, you have, again, you, it has to extrapolate. Yeah. Yeah. Yeah, and it has to extrapolate to the extremes or the sort of the, the black swan events. That-that's right. Yeah.

And so it's very hard. This is why things like self-driving cars are very, very-- It's a very difficult problem. Right. There's-- It's all corner cases. Yeah. It's kind of amazing how well they've done. That's an interesting point thinking about like when, you know, coming from the world of physics where a model was usually a, a, a single equation or a small number of equations which uniquely define a system and everything about it, and you just crank, you know, just find a solution to the system, and you've, you now know everything you need to know.

And I think something like AlphaFold was kind of a shift for a lot of people where before they thought, "Oh, protein folding is, you know, a problem where you just, if we find the right force field and we have the right computational engine- Mm-hmm... we will solve protein folding." And the thought of even really solving it in a data-driven way was only appeared a few years before, you know, AlphaFold, uh, AlphaFold 1 came out. And it's interesting that, uh, I think it sort of forced the new AI modeling has, I think, forced people to reeva-evaluate almost what is science because AlphaFold is, and similar models are incredibly powerful.

There's a lot of things that they've opened up as tools, but at their core, they oftentimes don't give intuition in nearly the same way that, let's say, the phys-- that most physicists historically would have wanted. And so I guess it's sort of there's this old saying, "All models are wrong, some are useful." Yes. Box. And yeah. Box said that. Yeah, yeah.

When do you find the data-driven models to be sufficient, and when do you want sort of like something which is interpretable that humans can actually understand? I think it boils down to almost like the difference between weather and climate. Mm-hmm. If you're in a data-rich regime, um, like weather or even proteins because of PDB, um, you can feel Oh, yes.

In other words, I've, I've got enough data to kind of cover, and so a statistical model like AlphaFold should do. Yeah. And so in fact, a lot of people happily use Alpha. It-- You know, I think it's really revolutionized my understanding. I'm not a biochemist, but people seem to love it. Mm-hmm. And the, and the...

And one amazing thing they did is they exhaustively, they just, just ran it on all PDB and, and published it, which is just really, really cool. Um- It's like six billion protein predictions or something. Yeah. So, so, so it- The vast majority are actually quite accurate. Yeah. Yeah. So that's just amazing. Mm-hmm. So but it feels closed, if you know what I mean. Yeah.

But when it's like climate, and it's open, and it's non-stationary, or you have to make these, uh, big extrapolations, you have to be much more cautious, uh, or maybe biology. Mm-hmm. Again, may-- and there's probably, there may be parts of biology like, "Oh, there was this, uh, virtual cell challenge from the Arc Institute." Uh- That had a funny result.

Uh, I, I know that people were-- there may have been some overfitting for at least that. Yeah. Wou-would say more, sorry. Or, um, I guess maybe at a high level, I think the simple baselines work basically- Oh, work very, very well. Things like- Yeah, yeah. Yeah. Yeah, yeah. Just like in the- It's one of, one of the classic things in whenever you do biology is just always start with a simple baseline. Maybe this is probably just good ML in general. So- It's good ML in general. Yeah.

[chuckles] Understand your simplest case in, in biology. There are many problems where they're extremely resistant to anything beyond the simple baseline, even if you have a lot of data. That, that's right. And in fact, I tell people the same thing. Yeah. I say always just, just fit linear regression. Just fit- Just do it. Just do it. Just do linear regression. Or an SVM. Yeah. Or SVMs.

I mean, SVMs are, are just a different, uh- Different form of- Linear regression. Yeah, yeah. Yes. So- Different basis, but- So when do you need the more process model-y thing? Mm-hmm. I think it's just when you, when you have, I mean, sort of climate- Mm-hmm... uh, is on one end, and I don't know, uh, m- weather maybe on the other end. Maybe-- Or that may be too extreme, but the, the...

I think it's just where are you on the, on the data richness thing. When do you, when can you feel like, "Oh no, I really have a closed, I really have a closed problem and, um, I think I can actually cover it"? Yeah, closed problem that you-- data fully covers is... Yeah, I think that, that makes a lot of sense in what I've seen as well. Mm-hmm. Yeah.

I think one thing I'm kind of curious about is when you're working on climate modeling, what are the... You, you talked about contrails, um, you've talked about, I guess, CO2, um, predictions. What are the, the, the broad things you're trying to accomplish? So one of them is- Oh. Mm-hmm... I guess making up interv-- like making interventions, and the other one might be, uh, making predictions for things like insurance or, um, like how do you help, like, adjust for some sort of climate change?

Or like, yeah, what, what are the, like, principal goals, I guess, for you specifically or the community at large? I think, you, you know, just like any community, there's probably many different goals. For me, I'm... or, and my team- Mm-hmm... uh, we're very, very interested in interventions. Um, um, so like which ones are, uh, possible at, at, at relative cost?

I mean, contrails was kind of amazing because it turns out the, the intervention's quite low cost, and we, um... And also one amazing thing about contrails is they are local, unlike things like CO2. Because so if a country decides to fix contrails over itself, it actually improves its... I, I mean, it, it has global effects- Mm-hmm... but it mostly improves the, the climate a little bit over themselves. Yeah.

So they, they like that. I, I guess, though, if you are in a cold climate and you want to warm it up, this is now your own- Oh. You could, oh. Yeah. It, it turns out that it's a little bit asymmetric. Uh-huh. The warming is constant essentially, and global. Yeah. The cooling only happens when you're sort of at a good, the sun is at a good angle over you.

So it's very rare that the uncertainty-- There are contrails that where our uncertainty bounds in terms of the warming, there are many, many contrails, mostly at night, of course, where the, where, where it's, it's like largely warming and we're, we're very sure in terms of like two sigma. Uh, there's very, not very many contrails where you say, "Oh, I know for sure that it's cooling and I want more of it," and like, like, like...

So only over sort of the poles, uh, in polar summer do you know that the contrails are cooling, and therefore if you got rid of them, they would warm up. But there are essentially no, uh, flights over Antarctica and not that many over, over the poles- Mm-hmm... uh, in the summer. So... So no one's gonna, no one who lives in a cold climate is gonna use this maliciously.

[chuckles] Well, uh, yes, in that, uh, in that... Well, they wouldn't know for sure whether it was warming or cooling- Ah... and so they would do stuff and things. I see. So mostly we just sort of ignore- Yeah... we, we don't recommend, uh, that people fly those. You also brought up a interesting point about the economics.

I mean, a lot of, uh, there I think was a lot of resistance historically about certain, you know, to climate, certain climate change interventions, which have in some sense the market has just taken over. Like at this point, unambiguously, like renewables and batteries are just almost universally unambiguously just better than alternatives. For, for- For, for- For, for, for non-mobile. I mean, you- Okay. Yeah, yeah...

for mobile, I mean, EVs are- Sorry. No, that's a really good point. Yeah, yeah. Like planes, we, we do not have a solution to- Correct. I mean, there are some battery-powered planes, but they're very s-small and have- Yeah... you know, very limited range. They probably will never actually be a or- It's hard to imagine. The physics would be very, very...

Un-unless we came up with something like nuclear batteries- Yeah... which would be kind of amazing, but I don't- Yeah... we don't know how to do that. [chuckles] Or even if we did, I think the, the risk of like people would be too afraid of a nuclear battery going wrong or something. Oh, yeah. It- Yeah... since we don't know what they are, we don't- Yeah...

know what the risks are. I guess they don't have the risk. Yeah. Yeah. So we don't know. So yeah, that's, that's the problem with I, I talk about, I have given talks about climate change- Mm-hmm... and I talk about the pi, the pie chart of badness. Mm-hmm. Pie chart of sadness, which is there's no one silver bullet for climate change, right?

There's so many different things that contribute greenhouse gases just from across our, uh, economy, so they sort of all have to be fixed or many, many of them have to be fixed. So there's no one single thing. I mean, I've worked on fusion. Fusion's cool, and it might actually knock a lot of them out- Yeah... if it's cheap enough, which we don't know, 'cause we don't know if it'll work yet.

I, I mean, fusion is one of those interesting things where the joke was always fusion is thirty years away, but I think it's actually now- Oh... less than thirty years away maybe. Yeah, no, I think there's a, there's a def-definite probability- Mm-hmm... that s- uh, that someone will make, uh, a commercially relevant- Mm-hmm... fusion even by the end of this decade. Yeah.

So I, I think it's like three years away, not- Yeah, yeah... not thirty years away. This is very real. Interestingly enough, I think a lot of that, um, I'm gonna now just And stump or, uh, advertise some of our other episodes, but a lot of that actually comes down to material science, um, interestingly enough, in that- Well, I'm, I'm skeptical...

well, superconductors. Um- Oh, superconductors. Yeah, yeah. Sure, sure, sure. Um- Well, yeah, sorry. Yeah. Okay. We can talk about fusion if you like, [laughs] but- Yeah, yeah. There was actually two inter-- two things. One is fusion, the other is, um, better control systems, which I think is actually a fair amount- Yes, and in fact, uh- Yeah... uh, Google DeepMind- Mm-hmm...

has been working on, uh, uh, control systems for tokamaks- Yeah... uh, uh, to make sure they don't, uh, uh- Disrupt... essentially go unstable and go disrupt. Yeah, yeah. Yeah, that's right. Yeah. Disruptions are, are quite interesting to themselves. [laughs] Yes. You know what? Yes. Yeah. It's basically the entire, the en-- all energy in the, the, uh, tokamak e-culminates into one little beam, and then- And, and hits your vacuum chamber- [laughs] It's-... and you're very, very sad... you're very sad.

[laughs] Yes. Yeah, yeah. I think people believe that ITER could be t- could turn on after thirty billion dollars disrupt and then basically have a thirty billion dollar, thirty billion dollar brick or something. Oh, I... Yeah, I guess. Yeah. I guess you could try to patch it. Yeah. I remember, um, uh, working, uh, we, we, uh, again, uh, before LLMs, we worked, uh, with, uh, a fusion company called TAE, and I was in their control room. And yes, it was kind of sad. You have to be very careful.

We were making systems to recommend new experiments, and, uh, they were very, very, uh, skeptical and jaundiced, which, where they should because, because I've even been there for the-- even under human control, it's like, uh, they were doing some experiment, and then you hear this big bang, and it was like, "Oh, no."

And then it's like, you know, then, then the apparatus is down for two weeks as they patch, so- [laughs] Wait, you were there during a disruption? Oh, no. This is- Oh, oh, oh... this is... Sorry, they have field reverse configuration. Oh, okay. So- Oh, got it. So- I see, yeah... which has its own, I mean- Mm-hmm... things, you know, there's some arc that happens. So, so what is that?

Sorry, I'm not familiar. Oh, oh, uh, what's the field reverse configuration? Mm-hmm. Well, it turns out tokamaks are not... Although they're perhaps the most studied form of plasma, there's many different kinds of architectures, essentially ways to try to stabilize and, and compress plasmas. Uh, there was a, a shape, essentially, it's essentially a self-contained football of plasma called a field reverse configuration, where essentially the magnetic field inside and outside are opposite, so they're separated by something called a separatrix.

And, uh, that is, sort of in theory, unstable, but in practice stable. Uh- Okay... like for example, when you run magnetohydrodynamics, MHD code, uh, it- it's unstable under that assumption, but that's, that's an assumption. That's not the way the real world works. And so yeah, it was kind of disfavored for many years. But, um, uh, TAE and other people, I think Helion, have, have, uh, FRCs, uh, because they are a- a- actually relatively robust.

Uh, you can actually knock them against walls, and they'll, they'll still, they stay stable. And yes, but you can still get discharges and things that punch holes in your vacuum chamber, which is kind of unfortunate. For clarification, so you have these fusion reactors. They are, or trying to be reactors maybe, and, uh- Mm, apparatuses, yeah. Yeah, apparatuses. [laughs] And you create a plasma. The plasma is magnetically- Con-...

charged. Oh, confined. Yes. Or confined. So it's confined by a magnetic field, so you have some sort of a magnetic system that is tunable by a computer, and then the computer tries to kind of maintain the- Well, in the-... the confinement? Well, FRCs kind of, once you make them, they're sort of sustained. Oh.

There's different ways of, of trying to make sure you... Okay, so all of fusion boils down to something called the Lawson criteria. There's essentially, uh, and it's, it explains why fusion is hard. Essentially, you can just very easily on the, on the back of an envelope just show that the density, the temperature, and essentially the energy loss, it's called the confinement time.

It's one over the amount of time it takes for the energy to decay away, uh, one over E in a plasma. So the th-- the product of those three numbers has to be bigger than some constant, and then you can get fusion, and if you don't, then you don't. And that's explains the, the fact that it's a product of three numbers explains why fusion is so hard because it, every approach has an Achilles heel where one of those numbers is not very big, and then they try to desperately make that be higher.

Yeah. Uh, and every approach is diff-- every approach to fusion is kind of different. And a lot of... You have to be a bit skeptical when there- there's all these sort of breathless news things about fusion because they'll say, "You know, now confinement time is starting to like, oh, stable for X minutes," or whatever it is, and it's talking about like one of the three numbers, but you have to have all three numbers before it, you can get fusion.

I, I think the, the, the whole field is, is making a lot of progress, and it's very exciting, but you do have to, you have to be a little bit cautious about the breathless news articles that only talk about one number. So what is the computational part of that? Oh, it unfortunately, for better or for worse, it depends on the approach.

So for tokamaks, as Brendan said, um, it's, uh, that there's this, it's mostly stable except that there's occasionally this instability that takes all the energy and max it, smacks it into one place, and so you have to sort of keep everything sort of under control. So it's a control system. Uh, FRCs themselves have very simple instabilities. So for example, they have what they call a Z instability. So it's fine, it's stable, it'll just wobble.

It'll literally wobble back and forth, but you just make what they call a PID controller that just keeps the football in the center of the reactor and, and things are fine. And it does that by adjusting the magnetic field? Yeah, it sort of... It actually just, I think the electric field is sort of by, sort of knock, knocks it back and forth.

The issue that people have is it really depends on which, which sort of plasma architecture they're deciding to use. Yeah. Climate is, you know, sort of political because of economics, basically, probably, mostly. Maybe other stuff, but the, the, the economics of it, you know, you have to persuade people to, to somehow spend more, or you have to have a solution that it ha- it has like this happy coincidence where it's be- both economically better and, and better for the climate.

That is hard. Yeah. Many, many- So, so but it- Many cases it's not. I mean- So-... many cases it has, hasn't been hard. But- Yeah... yeah, I guess. Yeah. Well, like when you think about like, okay, uh, predicting even weather, right? You can prep and you could see how that could be economically beneficial.

So what kind of work are you doing with interventions, and how does that kinda Um, interact with economics. Like, it sounds like the contrails one- W- -you guys did it in a, uh, an analysis and said, "Actually, this is great because it, it's very low economic impact, but high value." That's right. So if you're jo-- trying to think, there's, there's sort of energy intervention, so you have to be-- you have to sort of compete with existing forms of energy.

Um, and that, that's not trivial unless there's a co-benefit or there's some, there's some sort of clever, um, you know, just co-benefit. Like, uh, this again is highly speculative. It wasn't our work. There was a, a startup that was-- I don't know if you saw the news. It was last year, I think, where someone figured out if you inject, um, mercury into a, a fusion reactor, the, the neutron flux can actually transmute the mercury into gold.

[laughing] And then you, then you can sell the gold- [laughing] -uh, which I thought was very clever. It might not work, but I think- [laughing] You know, as, as a physicist, the one thing I want out of a fusion reactor is, is helium, but that's a different story. Sorry. Oh, you, you-- oh, he- Yeah...

helium, helium at three. Well, I mean- Yeah... helium four is kind of boring, although it's, it is getting-- because the strategic reserve has- Yeah... been shut down- Yeah, yeah... it's, there's less of it. Mm-hmm. Yes. Uh, and of course, I want helium-three and four. Helium three, yes. Well, not even just a fuse, just to make dilution refrigerators for quantum. Or MRIs or- Oh, yes... there's so mu-- yeah. Yes, yes, yes.

So much, so much technology we think about actually just goes out the window if we run out of helium. [chuckles] That's true. No one's thinking about it. Sorry, that's like a complete aside, though. [chuckles] Uh, yes. The fact that- Uh-huh... that the US had a helium, strategic helium was to reserve was for a very important blimp fleet. Mm-hmm. Yeah. [laughing] Uh, [chuckles] but, but they kept it for decades anyway. Yeah.

So that was nice, but- Yeah... then we stopped. Okay. And then we got rid of it. It all, it all went up in the air, literally. Yes, yes. In, in balloons and stuff. Yeah, yeah. Or out of natural gas wells. Yeah. Mm-hmm. Um, sorry, now we're talking about helium. Yeah, yeah. [laughing] No, so, so inter-interventions, what are some of the most exciting, interesting ones? Well, I'm very excited by fusion.

I mean, I don't know if that's intervention. That's sort of a s-- uh, uh, a source of energy. 'Cause if we can make, if we can make it work, and we can make it be sort of low enough capital cost, then that will actually, um, uh, help a lot because at least the, the current models are-- renewables are great.

Ideally, you'd like to electrify everything, right? Which has problems because, like, you can't electrify flights, but you can try to electrify a lot of stuff. You know, there are EVs. Um, you'd have to figure out how to electrify things like, uh, c-- uh, cement or steel. Th-those are hard, especially things like, um, uh, making steel want reduction power anyway to essentially, uh, you're adding carbon, and you're reducing iron ore.

Um, so there's a lot of sort of things that are difficult about electrifying everything, but if you could electrify everything, then it, then the amount of electricity required would grow by a factor of five, and you could try to grow, um, renewables. Renewables plus battery, again, trying to squeeze all of it out, it starts getting ever more expensive because you just need ever more-- you need, like, a huge number of batteries to sort of cover the, the last, you know, few percent or even, you know, ten or twenty percent.

So we do need some sort of power that, that, uh, can cover the last twenty percent, something that's, you know, base load. So, uh, fusion might be a, a, a thing for that. Um, so that's super exciting. Again, there, there's no one sort of silver bullet that can sort of cover all the cases, so I'm happy to sort of talk about any specific case, but it's sort of like- [chuckles]...

the world, the world is a very complicated place, and the- Yeah... the global economy is a very complicated place, so it's super hard to sort of talk about sort of interventions in general. Yeah. Maybe instead of interventions, one thing I'm curious about is how does this, uh, make, uh, affect decisions into, for example, like, where do we build? How do we build? Um, I think you're from LA, right?

Or at least you- Well, I spent eleven years there, yeah. Your life. Your li-- okay. So yeah, you spent a lot of your life in LA. I mean, LA just basically large parts of it just burned down, um, and maybe probably close to where you used to live. [chuckles] Um, so the, the, the, you know, this is something that I think a lot of people kind of saw coming, um, maybe partially due to regulatory issues, but partially due to other issues, you know, and we were completely unprepared.

And it seems like there is a, a lack of preparation about what to do next or to sort of adjust for this. And, I mean, have you w-worked on basically predicting, like, new risk assessments or, you know, suggestions, uh, like what do we actually change to maybe harden, you know, society even that-- for what's coming, regardless of whether or not we actually do something to make-- solve the underlying problem?

That's right. So, uh, in fact, there's a big effort at, at Google into something called cl-cr-crisis resilience. And so we had a very, uh, fun project called Fi-Fire Sat. I, I don't know if you know about this. Um, so it turns out that, um, uh, for wildfires, a lot of these wildfires you could ca-- if you, if you only caught them early enough- Mm-hmm... uh, it's very easy to put out a, a, a wildfire the size of this room.

But even if it's like an acre, it gets much, much harder. Mm-hmm. And so-- And of course, under certain circumstances, they can grow, you know, exponentially from, from the size of this room up to an acre, so that might be hard to catch. But they often sort of start small and spend a while. So, uh, we figured out that, oh, if you had a global constellation of low Earth orbit satellites that could, uh, detect in the mid-wave IR, that-- which goes back to the black body. Essentially- Mm-hmm...

that's the temperature of fire. Uh, they stand, the fire stand out, uh, in the mid-wave IR. Mm-hmm. And so, uh, we designed, uh, a sensor that if you built, it, it depends on exactly what their orbits, but you know, roughly fifty to eighty of them- Mm-hmm... you could actually find, um, fires about the size of this room, uh, uh, uh, about five meters on a s-- well, maybe it's a bit larger than this room, five meters on a side, um, and anywhere on the planet, and again, depending on how many satellites you had within, within, like, fifteen to twenty minutes.

You'd have to put, you know, a fair number up, like, you know, eighty to get them within fifteen minutes. Mm-hmm. And then you could actually intervene. You could decide not to if you wanted to, you know, uh, have the fire burn fuel, and you thought it was safe. But if it was going to blow up to something unsafe, so we worked with a, now a nonprofit called Earth Fire Alliance that we're part of, and so they're starting to lau-- we've launched one satellite, which is a prototype.

We've worked with a company named Muon Space to- Mm-hmm... actually sort of make the satellite, so that's cool. We have wildfire Boundary detection and we propagate that information out through Google. So, uh, we can actually sort of figure out from e-existing satellites and existing data feeds where the boundaries of, of fires are, and then we sort of tell people through, through their Android phones or through search about fires.

Um, we've worked with the, uh, US Forest Service on making new models for how fires propagate, 'cause again, that goes back to these process-based models from the '70s by a person named Rothermel. So we've actually made a little neural network, uh, proxy model based on a new-- So essentially to sort of be able to run it very, very quickly. Uh, so we worked with the Forest Service on that.

Uh, so yeah, yeah, we're very, very interested in trying to minimize... Because it turns out people might not realize the World Health Organization estimates that there are three hundred thousand excess deaths a year across the world from wildfire smoke. Yeah, I mean, I remember, um, was it-- If-- It's been a few years since we had a really bad fire season, maybe what, four, four or five years ago, there was this cloud of smoke which like crossed all of Northern US and Canada and caused a lot of respiratory issues, I think.

Yes. And, yeah. Yeah, and it's very hard to track. I mean, you have to get these, you have to get the, estimate these excess deaths from statistical means. Yeah. But yeah, it's a very serious public health problem, and also just very scary, and it burns people's houses down, and it's terrible. Yeah, yeah. My view is that climate change is sort of like a serious disease.

Do, do you treat the symptoms, i.e., do you adapt, or do you try to attack the underlying thing? And the answer is, well, if it's a serious enough disease- It's both... both. Yeah. [chuckles] Right. And so, yes, so- Mm-hmm... uh, we take sort of adaptation, especially around cl-climate resilience, very seriously at Google, and we try to give people informational tools to sort of help. Yeah.

That's part of the reason why we're working on weather, uh, and, and sort of cyclone prediction and things. So it all actually hangs together. So it's more than just, you're right, it's more than just interventions, it's, uh, climate resilience too. Yeah, having lived through four or five fire seasons on the, on the West Coast, they're, they can be quite, quite nasty.

And it used to not be, I mean, uh, I have a cabin up in the Sierra Nevada mountains, and yeah, it used to be, oh, you know, summertime, it's nice. And then now it's like, well, not every year, but yeah, there's like, you know, winter, spring, summer, and smoke. [laughs] Smoke. Yeah, fire season. You just- Yes. Yeah.

[chuckles] Uh, and- I wanna stay, stay to the west of the fire line and yeah. Uh, yes. And I- Yeah. Yes. So that is another thing that, uh, I'm i-interested in and that Google's also very interested in is, is climate, uh, resilience. Are these IR sensors small enough that they could hitch a ride in like a micro satellite grid? Like would it make sense to...

Would it be almost cheaper just to hire, I mean, to, to pay someone who's launching a constellation, um- Oh, they're not that small- They're not that small... 'cause they need-- The thing is you need, um, refrigeration because- Oh, okay, okay... because it's mid-wave IR. I see. So you have to, you have to keep it cool to- I see.

So these would have to be their own special satellites. They're not, they're not super large. Okay. Yeah. Um, they're not like the, the, the, you know, the, the satellites in, in geosynchronous orbit- Yeah, yeah... are these giant monsters because of all the optics and- Mm-hmm... who knows what, but- I see. Yeah... um, yeah.

And they have, you know, they're, they're basically just IR sensors with a resolution of five by five, um- Uh, no, no-... or something or- That's the other cute thing is their resolution is about fifty by fifty meters, but you can use super resolution because it's essentially multispectral, and you sort of know where fires are and you have-- So yeah, there's a, a fair sprinkling of AI on top of them to, to reach that five by five meter.

Oh, that's cool. Yeah. Yeah, yeah. When you also have, you have a convolution over the, what you're reading out, right, as well? The-- I forget the frame rate. Um, the satellite is moving. I don't remember what the point spread function is. I'm sorry. I don't- Okay. [laughs] Uh, uh, but you're right, they do move. Yeah. But I don't remember how fast they, I don't remember how fast they, uh...

This also has been called a broom sensor, so there's this funny thing of, of trying to, you kind of spread out the spectrum one way, and there's also, it's, it's a somewhat complicated thing. It isn't just a, like a, a Polaroid. It's, it's a complicated sensor. Yeah. Sort of switching gears a little bit, um, you know, you've been at the intersection of AI and science for quite some time. Mm-hmm.

I think you've sort of w-wound your way into and out of it back and forth. How do you see the field has evolved? Because I feel like it's evolving very quickly now, and like what are the sort of lessons that you've, you've learned, that you think the community has learned, and how do you think this should change, i-if you were a young scientist or young practitioner, how should this change what, how you should approach, you know, the future?

Well, I think there has been a phase change in the last twelve to eighteen months. So I mean, a lot of what we used to do, as I, as I said, was, um, uh, build, uh, these specialized models to sort of solve- Yeah... individual problems. And if you think that's your job, it's kind of fun. You, you- Mm-hmm... find a problem, you solve it, you find another problem, you solve it. Mm-hmm.

But now we have these much more general AI things, and I think the whole AI for science community is kind of still feeling around. They're so-- The, the fact that they're working is so new that collectively we're not sure, like, what's the best thing to do? Or maybe there's no one best, maybe there's a tool chain. Mm-hmm.

And I think, I think we're all trying to figure out, like, what should we do? And so there's a question of, uh, what should, uh, young scientists do? I think it would be, uh, you know, I, I ha- I have a son who's, uh, who just turned twenty-one, and he's r-really into both sort of AI and coding and chemistry.

And I look at him, I think he's doing the right thing because he's both learning a lot, uh, trying to be a domain expert about RNA, but he's also sort of using vibe coding and using all the tools. I think that's, I think that's the right answer is, is, is because everyone's figuring it out. Still be a deep domain.

I don't, I don't think domain expertise is going away 'cause it, it goes back to a lot of people have said it goes back to taste and trying to figure out how people get taste without doing all the, the grunt work. That's an interesting open question. But, but develop domain expertise, but also try and play, I would say, with all the different tools that are available because it's not like, "Oh, yes, we know what's gonna happen," and the smart old people are knowing what's...

Like, no, we're experimenting too. And so, uh, yeah. So I would say, uh- Definitely develop domain expertise and try to use these tools, and try to solve, uh, big, hard scientific problems as best you can. There's still the huge open issue about what do you actually do about physical lab work that is not going away because, you know, experiments are the, the ground truth, and- And a bottleneck. And a bottleneck.

I mean, people are talking about a lab in the loop, but that's still very, very, very open because how do you, how you... No one has, I think, as far as I know, a general lab that does everything. There's a lot of very specific, you know, labs that are controllable. Um, so I think it's just we've gone through this phase change. Uh, it seems super exciting.

Again, I would advise people to play with whatever tools are available and to develop sort of deep domain expertise and, and taste to the extent you can. And I would advise people also not, not to be scared and learn, just like try stuff, you know. Uh, you know, I'm always happy. Uh, we have student researchers at Google, and they come, and they do sort of wild and crazy things, and that's always just delightful. So yeah, people should be trying sort of wild and crazy things and see what happens.

Yeah. This may be a question without an answer, but when I think about how I developed expertise and how a lot of people developed expertise, it was by starting with a, you know, a, a simple, well-defined problem and then hammering it. And then in that process of exploration, you learn more and, you know, some ways you go broader, some ways you go deeper.

But you-- So the process of just banging your head against a problem, which now would be instantly solvable, teaches you the skills you need to solve harder problems. What advice would you give to your son for that? [chuckles] You know, I don't know. Maybe it's a bit like hiking, which is you... Yes, I mean, you obviously can't drive everywhere. You could, you could drive up the mountain- Yeah. Mm-hmm...

or you could hike up the mountain, and maybe it's okay, even fun to occasionally hike up the mountain, even if you can drive up the mountain. Uh, yeah, I mean, in the old days, I'm gonna sound like a real old man. In the old days, like- In the old days of, uh, six months ago. [laughs] Well, no, no, I was even thinking of in the old days of the '80s and '90s.

Like, you know, a lot of people take it like, "Oh, there's open source packages, there's, there's Sklearn, there's, there's all sorts of things." We didn't have that. I had to write my own numeric library. I had to write my own machine learning. I've, I've written boosting, probably rewritten it four times- [laughs]... in four different languages, and, and so now I know boosting. You know, it's like...

And so maybe- Mm-hmm... uh, not taking the totally easy path. Obviously, I mean, there's this trade-off like, "Oh, but I want to be as efficient and, and productive as possible." Yes, but you also have to develop the muscles. So it's a little bit maybe like being an athlete. Like there are places, there are times when you're actually doing exploit, when you're trying to run as fast as you can, and then there's also training time. And so maybe people just have to train.

And it's entirely plausible that if you spend time actually hammering away and doing the hard work, even if it goes slower there, that pays dividends into your larger, um, you know, productivity long term. Like even if locally, that one moment you are not being maximally productive by not exploiting an LLM, that feeds into something. I hope so. I hope... The thing I don't know is I hope people in their careers...

It's hard because, right, the, the, the whole world seems to want to optimize everything, and- [laughs] It's your fault. [laughs] No, I know. It's... No, I know it's my fault, but it's just sort of the, the, the, you know, uh... And sometimes you have to set aside time. Like at Google, especially in my group, we have this concept of twenty percent time, which I still, I very, very strongly in my own group try to protect. It's like, you can do whatever you...

If you wanna learn stuff, if you wanna try stuff, you don't even have to tell me. You don't even have to tell me. In fact, probably shouldn't tell me. [laughs] Uh, you know, just do stuff for, for exactly for learning and also because that's where the sort of creative juices are. I don't wanna so occupy people's time where they have nothing, like where they, they can't feel like they can play or learn or try new crazy things.

So I know twenty percent time is unusual, and there just seems to be this strong impetus in the world to just, like, like you said, optimize and squeeze everything out. But you do lose something when you hyper opt-- You sort of overfit. Yeah, you overfit. [laughs] Exactly. So- You overfit to productivity as you- Yes... so, so mentioned. That's right.

So I, I know that might be, my advice might be swimming upstream against, uh, perhaps cultural norms. The thing that always comes up for me here is that I've-- there's an-- The problem is not stationary. There's a new skill set that will be the, the right skill set for the future, right? Yep. And the, the question in my mind is always just detangling, okay, is this a skill that is an enduring skill that- Yes...

that or, or like maybe it, it wasn't enduring yesterday, but today it will be enduring because the... Like I've seen that, um, you know, like for just as an obvious one, you, you become sort of like a manager when you're- Yes... w-when you're using LL- Absolutely. I tell people this. A lot of those skills transfer well.

Some of them don't, but it's a lot of them do. And so that as a manager, you lose track of, of the details of what's going on, and you trust your people or agents or whatever to have that managed so that they can report up to you and answer, uh, you know, sort of the high-level questions and, and get the judgment about the little things correctly.

And, and so that, is that w-what we've come to? Is it we're just like middle managers now or like- [laughs] Well, I, I, I mean, I don't know. Again, I, I, I have, uh, a little management org, but you can't-- I don't-- You don't wanna be an empty suit. Yeah. In other words, because the things might get it wrong, especially LMs that are, that are sort of really weird. They don't- Yeah... make the same kind of mistakes that humans make.

And so you, you can't, you, you know, fully trust them. You have to be rigorous and like, you know, poke at it and make sure. Although you should be poking at software that you write yourself too. I mean, you shouldn't trust yourself. That's, that's one thing I've learned. What did Feynman say? You know, you absolutely can't fool, fool yourself, and you're the, you're the easiest person to fool.

You're the easiest person to fool. Yeah. So, um, so that might be enduring. I don't know if I, I can quantify what's enduring, but somehow fundamentalness, I mean, the, uh, really learning a domain that is about the world, for example. So this is why I like, uh, well, biology or physical sciences. I think those will... those are enduring sort of fundamental things. Math is very enduring.

But even things like rigor and checking and that sort of thing, which goes back to maybe management, that, that you want to really make sure that the LMs are Producing the right things or they haven't cheated in some way. But again, you should be doing that to yourself too [laughs]. Yeah. So I think there's some enduring-- And also just the enduring value of sort of creativity and thinking out of the box, and so I think those are enduring.

I don't know, there's some, there's something very fundamental about all of those. So you mentioned Feynman, um- I- If you don't mind me, uh, changing gears. Yeah. Yeah, yeah, yeah. I, I took a class from Feynman. Yes. Yeah, yeah, I was gonna ask. Well, yes. Yeah, not just any class. Yes, the, the, the, his, his sort of, uh, physics of computation class. Mm-hmm. He did it with, in fact, Hopfield and, and, and, uh, uh, Carver Mead.

Uh, that was fun. At the time, I don't think any-- maybe Feynman knew. I, I felt like none of us knew what even the problem was. I mean, I guess Feynman was trying to say, "Oh, let's do quantum simulation," which I guess is, uh, uh, turned out to be the right answer. But yes, at the time, it was-- Well, I guess, uh, it was cool.

First of all, it was fun, uh, DARPA funded it, so you were supposed to go once a week to get a prime rib dinner, uh, but I just went every week. [laughs] Uh, [laughs] anyway, for the- Yeah. For the students. Yeah. So ju-just for a little bit more context, this class was basically the class right after Feynman, and I forget who else, proposed the concept of a quantum computer without really knowing what it was, but knowing that there was some sort of- Well, there was plenty of- This is like- There was plenty of room at the bottom essay- Uh-huh.

Which I think was in the very early... This, I took it in eighty-two. Okay. Uh, that was when he taught it. And he did say there was plenty of room at the bottom. But the way the class was structured, it was, um, uh, it was like a guest lecture on Tuesday, and then Feynman would stand up on Thursday and explain why that was all wrong. [laughs] [laughs] Uh, which was pretty fun.

And then we encountered something which oth-other people have also encountered, something which is called the Feynman effect. Maybe he was so charismatic or something. He would explain things, and you would say, "Yes, yes, I understand." And that's the way it- And then you walk out thinking, "No, no, I, I don't understand any of this." [laughs] That did not stick. Yeah.

So it was kind of fun, but a lot of the guest people, I mean, maybe this sh-sort of showed the chaos, I mean, a lot of people, I think it was, uh, Danny Hillis came. Uh, there was all these interesting guest lectures. So I would say in the union of all of them, I think showed the sort of mass confusion of what was going on, 'cause there was a lot of like, "Oh, should we make computers reversible?

Because we have to make sure, you know, can they even be reversible? Can, can the sort of the bottom limit of, of the heat per operation be zero, or c-- is there some sort of thermodynamic limit?" And that was like a big deal. I don't think that's a, I don't think that's a big deal now. Yeah. But- Wait, so, so you're talking about the Landauer limit, right? Or no, what's that then?

Or- Uh, well, there-- at the time, it was maybe all this, this question about, like, can you have rever-- like billiard ball computers- Mm-hmm. And can they be reversible and things, so. Yeah. And also, it was sort of like what kind of computing, that's why Danny Hillis came, like, you know, I think that was in the era of the connection machine and what, what-- The original thinking machines, not, uh, Miran Marati's, but the original one in the eighties.

Mm-hmm. Uh, so he came. What was the state of, like, general computation in nineteen eighty-two? Oh. Like, I mean, I, I mean, at this point- Due to rounding error, we had zero computing [laughs]. Zero computing, yeah. I mean, we had-- I remember when I, sorry, I got there- Yeah. Uh, I was Carver Mead's, um, uh, sysadmin, and we had a VAX eleven seven fifty that maybe did a MIP, one million- [laughs] Ooh, one million operations, and the whole, uh, research group shared an eighty megabyte disk drive that was the size of a dishwasher.

It was, it was very exciting. Uh, sorry. So what about complexity theory? How, what, what was it? 'Cause I know that there's a lot of interest in quantum complexity and how it relates to gravity right now, and I wonder, I don't have a mind- The, the- My mind about weighing complexity theory. We, we, we could try to look it up. I don't think- Yeah. There's of course the whole, there's a whole hierarchy of, you're right, quantum complexity, uh, classes.

I think that was developed after that, 'cause this was in nineteen eighty-two. In fact, I, I'm not even sure there was a gate mo-- We never talked about the gate model. There was no gate model of quantum. You mean quantum gate? Yeah. Yeah. Yeah, yeah. Yeah. Yeah, yeah, I forget. I'm sorry. I, I forgot. I think a lot of those-- Yeah, I don't think those came out until, like, the, the what, nineties or something. Yeah.

Yeah. I mean, I, I remember reading Nielsen and Chuang, the classic quantum for... But I, I don't- Yeah, yeah, information. Yeah, quantum information textbook, which, uh, which brings up a lot of those points. Now that I think about it, I-- That textbook was written in, what, the late nineties or early two thousands. Yes, that's right.

And I think that was the first textbook which, like, put down kind of the general knowledge of the, the, the field. But I could be wrong. I guess I, I think now that when I was young, I don't think I, I took it just for granted that this is, this is the book that everyone- Yeah, no, but this is the book Was using for reference.

Well, you remember in the, in the early eighties, um, I mean, there was a whole bunch of excitement around neural networks. Mm-hmm. And it's really interesting 'cause, again- [laughs] We really did not know what we were doing. Yeah. No one knew what they were doing. There was, like, an interesting, like, neural networks, then SVMs, then neural networks. Oh, no, no. This is before that. Oh, okay, okay. This is in the eighties. Uh-huh.

Everyone said, everyone said, "This is, has the capability of revolutionizing computing, but what does?" Well, I mean, there was a tremendous, um, uh, excitement around Hopfield networks. Uh, uh, in fact, NeurIPS came out of, uh, because, uh, there was a workshop at Snowbird that was nominally private, but everyone tried to crash, and so they, they spotted up NeurIPS.

And so that was starting- So Snowbird is a, is a skiing trip with a, with a computation conference attached. That, that's right. Yeah. That's, that's why I learned to ski. Yeah, yeah. [laughs] 'Cause I didn't know how to ski, and then I kept going. Anyway, that workshop came out of the Santa Barbara workshop in nineteen eighty-five, which came out of some local things at Caltech, which was called Hop Fests. Right.

[laughs] So, so yeah, people thought, "Oh, wow, something involved..." In fact, ironically, there was like, "Yeah, yeah, something about associative memory." Mm-hmm. And if you actually dig down into what transformers are- Are They are associative memory. Are associative memory, yeah. So in fact, there was even a paper called, you know, uh, Hopfield Networks Are All You Need.

Uh, so yes- Oh, there, there was- Yeah, I forgot, I'm sorry. So was that a, that title was a reference to that paper? Uh, to the era. The era. To the era. Yeah. So it's actually, it's like- Oh, wow. I didn't know that It's like the whole thing has sort of come full circle. And in fact, I think we did, uh, we have, uh, um, collectively as a, as a field, uh, revolutionized computer science, but it was the, the sort of the hopes and dreams completely outstripped, uh, the capabilities because, again- Mm-hmm.

We effectively had zero compute. Yeah. I think there's, the, the history of machine learning is different paradigms as, like, compute versus memory, uh, scaling versus data become available in different- That, that's right. And the fact that I think people don't realize that the reason why neural networks won is they're the one compute-limited thing, although again- Mm-hmm...

with, we-- now that we have transformers, things are getting memory limited again. But, but-- and the fact that they ride on top of BLAS, and so the fact that BLAS was being optimized by things like GPUs, I mean, I don't actually know if... In fact, I'm pretty sure the brains don't work by matrix multiply [laughing] But, but it was just that, that it did, that the algorithms co-evolved with the, with the hardware. Mm-hmm.

And, and that's, that's why we're here. Who knows? I mean, if we'd gone down some other path where people really cared about some other compute- Yeah... who knows what, what architecture we would have ended up with? I don't know. I mean, didn't a lot of this start with like, I think people were hacking PlayStations to train, to, uh, train neural networks or something, or to do-- I guess su- maybe it was even before that, s-for like super- There's a, there's a funny story by, um, a friend of mine from grad school, uh, Brian Katzenzaro at- Mm-hmm...

NVIDIA, where he-- Brian, sorry if I get this wrong, but [chuckles] um, he came to NVIDIA and, um, and was trou- having a lot of trouble getting traction and, and, and he basically, they were, they were doubled and tripled down on, on gaming. And he basically, uh, one day had a meeting with Jensen and convinced him, "Let, let's..."

You know, like, "This is-- We have all these people using CUDA for BLAS," basically, and, and for deep learning in particular, and convinced Jensen. And he, he said it was like a fifteen-minute conversation, and they pivoted the whole company the next day or whatever. [laughs] Okay. This I, I've never worked in.

I mean, I have friends, uh, uh, uh, Dave Kirk, uh, who's the first chief scientist, and, and Bill Dally were both friends of mine from grad school. So yes, but I don't know the details of- Yeah... of way. Now, remember that people, um, uh, like in, in, um, Hinton's group, even in the around twenty ten, they were using GPUs to do the deep learning.

But, but even as, as a-- even in, I would say two thousand and seven, two thousand and eight, they weren't that much faster than CPUs. Again, they, they, they only started really exceeding-- In fact, I think this is not a coincidence, uh, in the era of, you know, the original ImageNet and some of the speech, some of the speech recognition stuff.

So that's-- I think that's not a coincidence that they really resurged when, when, when GPUs passed CPUs. So there's-- that, that's not a coincidence either. Yeah. I really wanna know, how does one get the opportunity to name an asteroid? Oh, well, again, is, uh, Caltech, uh, there was, uh, there was, uh, were some wonderful people, uh, Gene and Carolyn Shoemaker, and they were teaching a class in, um, in planetary science. I like planetary science.

And so yes, as part of that class, uh, they took us, uh, they, they took us through the sort of asteroid discovery thing. Now, this is in the eighties, so now there's all sorts of amazing, amazing, um, systems. Well, there was for a while, this something called Linear, uh, which automated the thing, but this was before anything was automated.

So yeah, it was just part of a class, and what you would do is you go the steps, even though we did it out of order in the class, but the steps we used to do, this was forty years ago, is you would go to Palomar. There would be a fast telescope, which literally now a museum piece that's in their visitor center, but at the time it was a real thing.

You would put a piece of film in it, you would take a picture, and then you'd wait for a few more minutes and take a same picture of the same point of sky. You'd put in a stereoscope, like back at Caltech. You would see if anything fl-- You would look around. You would see if anything floated, because it's this- Yeah... it would move by a tiny bit, so it would pop up, and you could see it was a solar system.

So you would-- Because it's different in different eyes than- Yes... you would be able to see it as a, uh, something that was projected in a different place in space. That's right. So you would see- Yeah... it would pop out at you literally. Yeah. And then you would go to a measuring microscope, and you would take measurements of known star references and where this floater is.

And so if you got accurate enough, uh, me- uh, measurement, then you would send it off to a person, I don't think it's him anymore. His name, Brian Marsden at the Minor Planet Center in, in Arizona. And then he had a big, um, uh, software system to do, um, to sort of piece together these called apparitions.

And if you found-- And if you happened to have done an observation, which was the final apparition, which allowed his software to connect it into one big orbit, then you would get discovery rights, and you could name the, the asteroid. Wow. But now it's like amazing. There's, uh, uh, there's this, uh, observatory now, uh, in Chile called the Vera Rubin Observatory, and there's this amazing telescope called the Simonyi Survey Telescope.

Essentially automates this process. Essentially can just take many frames of the sky. It's an utterly stunning instrument, and so they discovered eleven thousand asteroids in six weeks. Oh, wow. [chuckles] Yeah. So it's now- How many are them are there? I mean, at least of the- Oh, well, there's-- Well, it depends, I guess, on- You need that's the cut off... what, what the cutoff is. The cutoff size, yeah. But yeah.

So but there are probably millions of, of asteroids. So there's still a lot of opportunity. It's true, but I don't even know if they bother naming them anymore [laughing]. So I don't know. Maybe you don't name them. I don't know. Mm-hmm. But people are doing occult-- Sorry, I can-- I'll talk about this forever. Uh-huh. There's, there's some-- there's stuff called occultation. Like, one of my asteroids, I, I was, I was, uh, talking.

I was sending email to a amateur. Uh, if, if an asteroid happens to pass in front of a star, it, uh, dips. You-- It's like when they discover exoplanets. But if you're super lucky, it'll dip, and then it will dip again 'cause there's a moon. And so one of my asteroids, uh, this amateur, uh, found a little moon, uh, around it. So, uh, so that was I think last year.

Oh, around the asteroid? Yeah. Ah, interesting. So that's pretty cool. So you- Yeah... there's still room for, for- Your, your asteroid has a moon to it. Yes. In fact, apparently, it's actually, actually, there's a little bit of a story there because it's not really my asteroid that I found two of them. I named one of them after my dad.

I waffled, so Carolyn named it after a professor at University of Washington, and I said, "Oh, but I wanted to name it," and so I whined, and, and she was nice, and so she gave me one of hers, and so then they named that one after my mom. Ah. And that's the one that has the moon. Oh, okay. So, okay. So, so the asteroid named after my mom- Your mom... has a moon. How big is it? It's like the main body they think is about four kilometers across.

I'm trying to get the units right. Oh. And the moon is about one kilometer. So it's actually-- It's not-- It's a pretty big binary. They're not quite twins, but yeah. Did other people in that class Discovered, uh- Down your history, discover any... I think there was one other person. It's a little bit of a crapshoot. Yeah. And the fact that I found two is quite unusual.

Was there a reason why you... Was it just pure luck or- It was just pure luck... were you, like, just sitting there all day do- trying to- Oh, no, I was doing... No, I, I tried to be careful, but- Yeah. Yeah... yeah. How does one become, uh, get an Oscar? Yeah, an Academy Award. [laughs] Uh, well, again, maybe I was at the right place.

Uh, I, uh, my thesis advisor's name, Al Barr, and, um, I was a student intern actually in 1986 at a place called Schlumberger, which is a oil discovery place- Mm-hmm... but they had an AI lab. And so they were-- we were all doing sort of computer graphics research, and, uh, we were all thinking, you know, there-- At the time, this was revolutionary. I, I realize it's now considered incredibly boring.

Like, you could actually use, like, physics simulators to, to make computer graphics movies. And at the time, it was like, "Wow, that's really cool." So I said, "You know, you could use the theory of elasticity to make floppy things," and, and so I said, well, I've sort of like, "Well, here's the theory of elasticity," and I wrote a, a elastic simulator, and I made, you know, fabric and, you know, stretchy things and, and stuff.

So they said, they, "Oh, wow, that's cool." And so you sort of descendants of that became a lot of the physics simulators that people used in, in, um, you know, Pixar and their various movies. So yes, I tell p- I tell interns sort of half-jokingly, "Well, if you do a really good job as an intern, you can get- You can get [laughs]... you can get a Academy Award."

[laughs] So, so how, like, w- how long was that between the time you did that work and then- Twenty years It was twenty years, yeah. Which is actually not atypical, right? Uh-huh. 'Cause they wanna, when they give you an Academy Award, they wanna make sure, like, oh, yeah, it's sort of well used and everyone uses it and stuff. So yeah.

But, but of course, the ca- obviously, like, in that twenty years, like, well, everyone does it, it's obvious, but, you know. Yeah. This was specific work done for a specific movie or something? No, no. Okay. Okay. It was like a, it was like a paper in SIGGRAPH. It was, it was, it was a paper, and then Pixar just became, like, I think, somewhat important core to, like, a lot of their- Yes.

And al- and in fact, some of my friends, in fact, a lot of my friends did, did, did a lot of these simulators. So yeah. Mm-hmm. I'm curious about, since you have been quantum computing adjacent and worked on quantum computing directly at Google Applied Sciences for a while, I think you're not currently on that, but I'm, I'm curious to see what do you- I, I'm still dabbling in quantum error- Oh, you still are, you're still dabbling.

Yeah, yeah. In, in quantum error correction, but- Yeah, yeah. And yeah. So where do you see the trajectory of quantum computing over the, over the years going? Because this is one of those things which I guess kind of like fusion was sort of had a, a sense at first, like, being very exciting and then seeming like it wasn't going anywhere for a long time, and then maybe now we're seeing hints again of it being exciting.

And I, I'm-- Or maybe that's, like, my sort of quantum adjacent take on it or- No, no, I think a lot of people have gone through that. I tend to average, I tend to average things out over the dec- Yeah... I guess I'm old enough now where I sort of average things out over the decades- Yeah, yeah, yeah... and it's just, um, uh, progress.

But, but I mean, like, see, going from, you know, starting from Feynman just trying to figure out even what this means as a concept all the way up to now where, you know, I g- there's this recent Willow result of quantum error correction, which actually seems genuinely- Achievable... like the right, achievable with, like, the right scaling laws and stuff. Mm-hmm. I mean, does this, is this like a...

Do you still think that, or, I mean, would you say that quantum is twenty years away or, I mean, till, like, some simple but practical algorithm which actually succeeds? Or, I mean, is it, do you see this accelerating, or do you think this is still going to be something that's slog? 'Cause I mean, there's a lot of areas where we, we saw, oh, th-this is very, you know, th-this advanced very quickly, and it was very unexpected.

And I'm wondering if this is a thing that you think will be quick or will not be or is this, I don't know. I think sort of in between. It's not a purely software thing 'cause we need to build, uh, systems that are large enough and stable enough to- Yeah... be able to, you know, be a quantum computer instead of a quantum apparatus.

We're in the, what they call the NISQ era. Mm-hmm. Uh, that was a, uh- Noisy inter-intermediate scale quantum... intermediate scale quantum, uh, which was, I think, coined by John Preskill. It's not a very good term, sorry. Yeah. But I guess that's the term we have. Um, and there are, you know, uh, the quantum team made this, uh, wonderful, um, paper, which I think I'm co-author on, uh, one of many, for something called the Quantum Echoes, uh, algorithm, which could, uh, be applicable now, which fundamentally it's, it's, it's in the style of, of Feynman's, um, proposal, uh, which essentially it's...

Well, the technical term is you could try to fit a Hamiltonian to observe data like in an NMR. Mm-hmm. What that means is, yeah, you have, um, a physical model that's parameterized, and you use the, you use the, uh, quantum computer to kind of adjust the parameters and try to figure out in-inside of a loop to what the right...

So you could, for example, decode- Mm-hmm... uh, NMR parameters. So that is, uh, a practical thing that's, uh, used now. The question is, is it gonna be, it will be big enough to, uh, make breakthroughs? That's still, uh, TBD. Um, the, uh, quantum team at Google is, has been very, uh, executing, uh, again, amazingly well against a, a, a roadmap that, that, uh, Hartmut Neven, uh, laid out a few years ago, and they're just, uh, continuing to march down this thing where they're- Mm-hmm...

scaling up. And when they hit their last milestone, they should be able to have a, a quantum computer that does amazing things, and they're continuing to march it along. So yeah, it's, it's, I think on the scale of a few to several years. I, I don't, I haven't sort of kept up on exactly what date they're saying, so you should ask Hartmut exactly when that's gonna happen.

But, um, yeah, they're, they're, so they're marching along. So the, so the main, the interesting question is, will the superconducting, uh, computing be the winner, or will one of the other sort of alternate technologies, and that's still TBD. I th- I still think superconducting is, is a very promising thing because it is, it is very scalable. Um, so yeah. No, I don't think it's thirty years away. I don't think it's tomorrow. I don't think there's gonna be...

I don't think, unless... Well, even if there's a sudden hardware breakthrough, these are very finicky things, so it's like someone might have a brilliant idea, but it'll still be a while before. 'Cause fundamentally, at least in the NISQ era or for a while, these are fundamentally analog computers, and so that they tend to be very, very, very finicky.

So I wouldn't expect, um, all of a sudden, you know, something, some, some phase change- Mm-hmm... happens. No one has figured out how to scale up qubits in a way that interact in Just to kind of arbitrary size or still on the order of, of hundred, two hundred cubits I think, or- Uh, you mean in terms of- Or- Okay, so in t- In terms of- Well, so the difference- Or logical or physical, yeah...

so it's a little like, yeah, yeah, exactly. Yeah. So for superconducting qubits, um, yeah, people, I mean, although people are working on it, the ratio of the number of physical qubits to logical qubits is still relatively large for the error rates that you need. And so maybe there'll be a breakthrough there, I don't know.

Or there are other things that might have, uh, you don't need that ratio to be so high because a lot of it has to do with the, the two D connectivity of the, of the, of the chips that you lay out your qubits on a 2D chip. And for things like, um, neutral atoms, they in theory, in theory can connect anything to anything else.

But of course, in practice we don't really know, and we don't actually know what the limitations of neutral, at least I don't know the limitations of what neutral atom maybe- Yeah... maybe the super experts know. Superconducting qubits in particular have a sort of tension where you, you want to have, you know, nice clean resonators, which, you know, you do get a clean resonator by decoupling from the environment, and then you get good interactions by coupling resonators, which involves coupling to the environment.

So it always seems like there's a trade-off there. But- It, it's true, but at least there, but there, but there... Yes. There's no- But there's like the whole environment. Yeah, yeah. Uh, and then there's like little tiny peepholes- There's a piece of environment, yeah, yeah... that you, that you wanna go through to your neighbors.

This is not like a fundamental uncertainty relationship or something [chuckles] where- That's right... it's like this is a technological limitation that difficulty or something, but yeah. Yeah. Yeah. The hardware team in, in, in Google Quantum is, is very, very skilled. Mm-hmm. They're very, very skilled. Yeah.

So, um, um, yeah, they're, they're really good at, at making these designs and making these things actually work. Yeah. So I, I, I find them impressive. Yeah. It'd be exciting to see that advance. Yeah. Yeah, yeah. Yeah. Um, I guess I'll just have to hold a breath and wait. Just wait. Yeah. I guess I'm just very patient, so- Yeah [laughs]...

I've, we just start working- You said tw-twenty years ahead of, you know- That, that's what Dave, Dave Bacon, who runs the- Uh-huh... the, the software team in Google Quantum, always he, he teases me like, "John," like, "Oh, no, I can't work in quantum computing." This was like ten years ago "Because it's gonna be...

You're always twenty years ahead of time, and so I'm gonna have to wait for twenty years." Like, okay, well, you know, ten years have gone by. Ten years [laughs] It's like, you know [laughs] Halfway there. Halfway there. I don't, I don't take that as a, I, an absolute- There's a hard rule [laughs] Yeah. But, uh- Cool, cool... I also started working on Fusion ten years ago. We'll see, huh [laughs] May-maybe you're actually causal, like you start working on something and then reality just, you know- Catches up?... catches up. Yeah. I guess so. Maybe, who knows? I don't know.

Yeah [laughs] I, I started working-- I loved convolutional nets in the early nineties. Mm-hmm. And, uh, Yann, uh, LeCun claimed I, I- Is that how it checks out?... I coined the term convolutional net. Uh- Really? And as far as I can tell, that, that may be true. I... 'Cause everyone called it LeNet- Mm-hmm... 'cause it was a very specific thing, and they all worked for Yann. I said, "Well, I don't work for Yann.

I don't wanna call it LeNet." [laughs] So I called it a convolutional net. So I don't know. It was a more generic term. It's a good term. Yeah. Before you go, is there anything you want the audience to take away? Any messages you wanna deliver? Oh, I think Era is an example of it, but I, I'm really amazingly excited about the potential of AI for science.

I think it's, I think it's gonna be a, an amazing power tool for scientists, and I think scientists won't be replaced. I think it, uh, in fact, uh, I'm hoping that they'll spend all their time on, again, the creative stuff, on the rigorous stuff, on the philosophy stuff, and so I, I think it's gonna, I think it's gonna be way cool.

I, I hope so too [laughs] Yeah. No, I, I, yeah, I... My intuition is that a lot of, a lot of-- I've heard a lot of people say that, that, I hope they're right, and I hope it's not just, uh, sort of coping with [laughs] a reality that's uncomfortable. The other question we almost forgot to ask, if you could remove a bottleneck in your industry, which, um, whatever you, however you wanna define that by fiat- Mm-hmm... then- Like magic...

yeah, by, by magic, what would that be? If I could get a magic wish, I would say, "Someone please make, uh, the, the everything lab that you could like send JSON blob to, and it will do any experiment at all." Okay. Automated. But it'd have to be anything. Yeah. So essentially, so I guess we have to solve the sort of AI complete robotics problem- [chuckles] Yeah... I guess.

But if we did, then that would be stunning because right now things like Era, it's all computational. Right. So someone has to gather the data. Yeah. So yeah, if we could just break that, oh, oh, that would be so amazing. That would be so utterly amazing. Yeah. Um, great. So I really appreciate you taking the time to, to, to see us.

Um, and I think you, you like kind of flew in and adjusted your schedule a little bit to- Yeah, I was, I was sort of flying over San Francisco- Okay [laughs]... to head home, and so I just said I landed in San Francisco to go. So you, you made a big effort to be here. We really appreciate that. It was really fun to talk to you. Mm-hmm. Yeah. Well, it's been a blast. Yeah.

Okay, cool. Yeah. Thank you for having me. Yeah. You're welcome. Thank you. [outro jingle]

🔬 An Oscar, Two Asteroids, and the Algorithm in Your sklearn: John Platt on AI for Science

Kapitelklicken springt direkt hin

Shownotes

Transkriptvom Podcast bereitgestellt · Zeitmarken springen hin