🔬 Beyond AlphaFold: How Boltz is Open-Sourcing the Future of Drug Discovery

0:00 Actually, we only trained the big model once. Uh, that's how much compute we had. We could only train it once, and so, like, while the model was training, we were, like, finding bugs left and right. Yeah. [laughs] Uh, a lot of them that I wrote. [laughs] And, like, I would-- [laughs] I remember, like, us, like, sort of like, you know, doing, like, surgery in the middle, like stopping the run, making the fix, like relaunching, and, um, yeah, we never actually went back to the start.

0:23 We just, like, kept training it with, like, the bug fixes along the way. Right. [laughs] Uh, which was, uh- It was impossible- Yeah... to reproduce now. [laughs] Yeah. Yeah, yeah, no, that model is, like, has gone through such a curriculum that, [laughs] you know, it's learned some weird stuff. Uh, but, uh, yeah, somehow by miracle it worked out.

0:38 It's a pleasure to have with us today Gabriella Corso and Jeremy Wolven. They are-- they recently founded Volt, a company trying to democratize and bring our structure prediction in biology to, you know, the masses. Uh, they are both, uh, recent PhD grads from MIT and have been working on all sorts of foundational papers in, like, generative biology. Um, anyway, uh, pleasure to have you here. Thanks for coming. Thank you. Thank you.

1:06 Uh, I guess we're maybe, what, six years post AlphaFold 2 right now, which was, like, kind of a big moment. Is that right? I think it was, uh, twenty twenty-one. Yeah. So yeah, on- going on five years. Five years. Yeah. Five years, yeah. Yeah, yeah. Yeah, so maybe for the audience, like, let's go back to that moment in time and explain, like, what was this big moment and why was it interesting? Why was everyone so excited?

1:29 And I think you two were probably quite excited, so why were you personally excited? I would start on kind of why that was interesting, kind of, you know, from a scientific standpoint. So w- AlphaFold-- So maybe first as a kind of introduction for, uh, the ones in the audience who are not structural biologists. So the idea of structural biology is that, you know, we want to try to understand how, you know, proteins and other molecules take shape inside our cells and, you know, how they interact.

2:00 And structural biology is sort of this beautiful discipline, uh, where we are somehow able to understand this minuscule structure at, you know, kind of atomic details using, uh, these incredibly, um, complex methods like, you know, X-ray crystallography. And, you know, the, the dream has always been of computational biology, can we understand kind of the structures without having to, you know, resolve this crystal, you know, shoot X-rays and so on.

2:31 And so AlphaFold was a real breakthrough in this problem of protein folding, which is trying to understand the structure of a single, uh, protein. And to me, it was exciting across kind of many dimensions. One, I was computer scientist, I was working a lot on machine learning, and I saw kind of the impact that kind of the work similar, somewhat similar to what I was doing could have on, like, a long-standing scientific problem.

3:02 And on the second perspective, from a more, you know, personal side, the seeing kind of the structures coming out of these models where, you know, you see kind of this beautiful, you know, creation of life is something that was, was very inspiring to me. And so that was kind of one of the things that led me to start, uh, working on, uh, structural biology and in particular with machine learning. Were you a structural biologist before AlphaFold came out?

3:29 I mean, did you-- You did machine learning, but it was not in structural biology, so that actually shifted your career quite dramatically. Yeah, very dramatically. I was, I was working on some pretty kind of theoretical methodological things, and I was starting to see kind of, you know, some of the challenges in, you know, kind of doing somewhat theoretical or methodological work and, you know, seeing kind of the potential impact of, you know, um, doing excellent, you know, AlphaFold was really a machine learning breakthrough, but, you know, an applied machine learning and so that led me to, uh, want to start working in applied ML.

4:06 Our, our group at the time was, um, working a lot on like small molecules already, and I think AlphaFold is kind of what triggered, I think, this shift to like working on, on biologics. And at the time, I think it, like, opened as many questions, you know, as it answered in a sense. Like, we-- the immediate follow-ups were, "Okay, like, can we do this on other things than proteins?

4:25 Can we do, you know, interactions of small molecules with proteins, nucleic acid with proteins? Can we model more complex protein systems?" And I think, yeah, very rapidly, I think after AlphaFold, people realized, I think, that there was, you know, machine learning could have a-- could really, yeah, sort of target this problem very differently than, you know, than previous methodologies. Going back to the AlphaFold 2 moment, like, um, I remember this very well.

4:53 I was at NeurIPS when I guess the results of this famous competition came out. So you want to talk about CASP and, like, what it is and why it was so interesting and exciting? I think every-- So, every couple of years, I think the goal has always been to, you know, find protein structures that are a little bit different from what's known.

5:15 So CASP over the years has like, you know, put in a lot of effort to like gather structures from, you know, academic groups and, uh, even industry groups to try to create sort of a test set that would be difficult for, uh, different methods. And CASP, uh, 14 was when, uh, AlphaFold 2 really blew everything out of the water. The improvement was so large over, you know, the previous, previous method and also over the previous competitions.

5:45 And now CASP continues. You know, we've had CASP 15, we have CASP 16, and, you know, sort of what's happened now is that it's really expanding to also all these other modalities, like I was mentioning, like proteins, small molecule, nucleic acid, and-- But the goal remains to like, you know, really challenge the models.

6:01 Like how well do these models generalize? And, you know, we've seen in some of the latest CASP competitions, like while we're Become really, really good at proteins, especially monomeric proteins. Um, you know, other modalities still remain pretty difficult. And that's really essential, you know, in the field that there are, like, these efforts to gather, um, you know, benchmarks that, that are challenging.

6:22 So keeps us in line, you know, about- [laughs]... what the models can do or not. Yeah. Yeah. It's interesting you say that. Like, in some sense, CAS, you know, at Cas14, a problem was solved and, like, pretty comprehensively, right? But at the same time, it was really only the beginning. So can you explain, like, what was the specific problem which argue was solved and then, like, you know, what is remaining, which is probably quite open?

6:48 [chuckles] I think, I think we'll, we'll steer away from the term solved because- [laughs]... we have many friends in community who get pretty upset at that word, and I think, you know, fairly so. But the problem that was, you know, that a lot of progress was made on- [laughs]... was [chuckles] the ability to predict the structure of single chain proteins.

7:07 So proteins can, like, be composed of many chains, and single chain proteins are, you know, just a single sequence of amino acids and, uh, one of the reason that we've been able to make such progress is also because we take a lot of, uh, hints from evolution. So the way the models work is that, you know, they sort of decode a lot of hints that comes from evolutionary landscapes.

7:28 So if you have, like, you know, some protein in an animal and you go find the, uh, similar protein across, like, you know, different organisms, uh, you might find different mutations in them. And as it turns out, if you, uh, take a lot of these sequences together and you analyze them, you see that some positions in the sequence tend to evolve at the same time as other positions in the sequence, sort of this, like, correlation between different positions.

7:54 And, um, in-- It turns out that that is, uh, typically a hint that these two positions are close in three-dimension. So part of the, you know, part of the breakthrough has been, like, our ability to also decode that very, very effectively. But what it implies also is that in absence of that co-evolutionary landscape, the models don't quite perform as well.

8:16 And so, you know, I think when that information is available, maybe one could say, you know, the, the problem is, like, somewhat solved from the perspective of structure prediction. When it isn't, it's, it's much more challenging. And I think it's also worth also differentiating the... Sometimes we confound a little bit structure prediction and folding.

8:32 Folding is the more complex process of actually understanding, like, how it goes from, like, this disordered state into, like, a structured, like, state. And that I don't think we've made that much progress on. But the idea of, like, yeah, going straight to the answer, uh, we've become, uh, pretty good at. So there's this protein that is, like, just a long chain, and it folds up. Yeah.

8:55 And, and so we're good at getting from that long chain in whatever form it was originally to the thing, but we don't know how it necessarily gets to that state. Yeah. And there might be intermediate states that it's in sometimes that we're not aware of. That's right. And that relates also to, like, you know, our general ability to model, like the different...

9:16 You know, proteins are not static. They move. They take different, uh, shapes based on their energy states, and I think we are also not that good at understanding the different states that the protein can be in and at what frequency, what probability. Yeah. So, uh, I think the two problems are quite related in some ways.

9:35 Still a lot to solve, but I think it was very surprising at the time, you know, that even with these evolutionary hints that we were able to, you know, to make such, such dramatic progress. So I wanna ask why does the intermediate states matter? But first, I kinda wanna understand why do we care what proteins are shaped like? Yeah. I mean, the proteins are kind of the machines of, uh, our body.

10:01 You know, the way that all the processes that we have in our cells, you know, work is typically through proteins, sometimes other molecules sort of intermediate, you know, interactions, and through that interactions, we have all sorts of cell functions. And so when we try to understand, you know, a lot of biology, how our body works, how disease work, we often try to boil it down to, okay, what is going right in case of, you know, our normal biological function, and what is going wrong in case of the disease state.

10:35 And we boil it down to kind of, you know, proteins and kind of other molecules and their interaction. And so when we, we try predicting the structure of proteins, it's critical to, you know, have an understanding of kind of those, those interaction. It's a bit like seeing the difference between having kind of a list of parts that you would put in, in a car and seeing kind of the car in its final form. Right.

11:01 You know, seeing the car really helps you understand what it does. Yeah. On the other hand, kind of going to your question of, you know, why do we care about, you know, how the protein folds or, you know, how the car is made to some extent is that, you know, sometimes when it-- something goes wrong, you know, there are, you know, cases of, you know, proteins misfolding in some diseases and so on.

11:25 If we don't understand this folding process, we don't really know how to, uh, intervene. There's this nice line in the, um, I think it's in the AlphaFold 2 manuscript where they sort of discuss also, like, why we even hopeful that we can target the problem in the first place, and then this, this notion that, like, well, for proteins that fold, the folding process is almost instantaneous.

11:48 Mm-hmm. Which is a strong, like, you know, signal that, like, yeah, like we sh- we might be able to predict that this very, like, constrained, uh, thing that, that the protein does and so quickly. And of course, that's not the case for, you know, for, for all proteins, and there's a lot of, like, really interesting mechanisms in the cells.

12:05 But yeah, I remember reading that and thought, "Yeah, that's somewhat of an insightful point." I think one of the interesting things about the protein folding problem is that it used to be actually studied and part of the reason why people thought it was impossible, it used to be studied as, as kind of like a classical example of like an MP problem.

12:24 Uh, like there are so many different, you know, type of, you know, shapes that, you know, these amino acids could take. And so this grows combinatorially with the size of the sequence. And so there used to be kind of a lot of actually kind of more theoretical computer science thinking about and studying protein, problem, protein folding as an MP problem.

12:47 And so it was very surprising also from that perspective, kind of seeing machine learning so clear there is some, you know, signal in those sequences through evolution, but also through kind of other things that, you know, us as humans, we're probably not really able to, uh, to understand, but that these models have, have learned.

13:07 And so Andrew White, we were talking to him a, a, a few weeks ago, and he said that he was following the development of this and that there were actually ASICs that were developed just to solve this problem. So again, that there were many, many, many millions of computational hours spent trying to solve this problem before AlphaFold.

13:27 And just to be clear, one thing that you mentioned was that there's this kind of co-evolution of mutations and that you see this again and again in different species. So explain why does that give us a good hint that they're close by to each other? Yeah. Um, like think of it this way that, you know, if I have, you know, some amino acid that mutates, it's gonna impact everything around it, right? In three dimensions. Yeah.

13:52 And so it's almost like the protein through several probably random mutations in evolution, like, you know, e-ends up sort of figuring out that this other amino acid needs to change as well for the structure to be conserved. Got it. Uh, so this whole principle is that the structure is probably largely conserved, you know, because there's this function associated with it.

14:12 And so it's really sort of like different positions compensating for, for each other. I see. Those hints in aggregate give us a, a lot of information about what is close to each other and then you can start to look at what kinds of folds are possible given the structure, and then what is the end state, and therefore you can make a lot of inferences about- Yeah...

14:32 what the actual total shape is. Yeah, that's right. It's almost like, you know, you have this big like three-dimensional valley, you know, where you're sort of trying to find like these like low energy states. Yeah. And there's so much to search through that's almost overwhelming. But these hints, they sort of maybe put you in an area of the space that's already like kind of close to the solution, maybe not quite there yet.

14:55 And, and there's always this question of like how much physics are these models learning, you know, versus like just pure like statistics. And like I think one of the thing, at least I believe, is that once you're in that sort of approximate area of the solution space, then the models have like some understanding, you know, of how to get you to like, you know, the lower energy, uh, low energy state.

15:18 And so maybe you have some, some light understanding of physics, but maybe not quite enough, you know, to know how to like navigate the whole space well. Right. Okay. So we need to give it these hints to like kind of get there. So you get it into the right valley and then it finds the, the minimum or something. Yeah. Yeah. Yeah.

15:32 One interesting explanation about how AlphaFold 3 works that I think it's quite insightful, of course, doesn't cover kind of the entirety of, of what AlphaFold does, that is, um, that I'm gonna borrow from, uh, Sergey Chilingarov at MIT. So he sees kind of AlphaFold and the interesting thing about AlphaFold is it's got this very peculiar architecture that we have since, you know, used, and this architecture operates on this, you know, pairwise contacts between amino acids.

16:00 And so the idea is that probably the MSA gives you this first hint about what potential amino acids are close to each other. MSA is Mul- Multiple sequence alignment. Exactly. This evolutionary- What he was talking about. Exactly. This evolutionary information. Yeah. And, you know, from this evolutionary information about potential contacts then is almost as if the model is sort of running some kind of, you know, DiCE algorithm where it's sort of decoding, "Okay, these have to be close.

16:26 Okay, then if these are close and this is connected to this, then this has to be somewhat close." And so you decode, uh, this, that becomes basically a pairwise kind of distance matrix, and then from this rough pairwise distance matrix, you decode kind of the actual potential structure. Interesting. So there's kind of two different things going on in the, the, the kind of coarse grain and then the fine grain optimizations. Exactly. Interesting. Yeah. Very cool. Yeah.

16:54 You mentioned AlphaFold 3, so maybe a good time to move on to that. So yeah, AlphaFold 2 came out and it was like, I think fairly groundbreaking for this field. Everyone got very excited. A few years later, AlphaFold 3 came out and, um, maybe for some more history, like what were the advancements in AlphaFold 3?

17:11 And then I think maybe we'll, after that we'll talk a bit about the, uh, sort of how it connects to BOLTs, but anyway. Yeah. So after AlphaFold 2 came out, you know, Jeremy and I got into the field and with many others, you know, the clear problem that, you know, uh, was, you know, obvious after that was, okay, now we can do individual chains.

17:30 Mm-hmm. Can we do interactions? Interaction different proteins, proteins with small molecules- Mm-hmm... proteins with other, other molecules. And so- So, so quick, why, why are interactions important? Interactions are important because to some extent that's kind of the way that, you know, these machines that, you know, these proteins have a function.

17:49 You know, the function comes by the way that they interact with other proteins and other, uh, molecules. Actually, in the first place, you know, the individual machines are often, as Jeremy was mentioning, not made of a single chain, but they're made of the multiple chains, and then these multiple chains interact with other molecules to give, uh, the function to, uh, those.

18:09 And on the other hand, you know, when we try to intervene of these interactions, think about like a disease, think about like a biosensor or many other ways we are trying to design molecules or proteins that interact in a particular way with what we would call a target protein or target, um- You know, this problem after AlphaFold 2, you know, became clear kind of one of the biggest problems in the field to, to solve many groups, including kind of ours and others, you know, started making some kind of contributions to this problem of trying to model these interactions.

18:44 And AlphaFold 3 was, you know, was a significant advancement on the problem of modeling interactions. And one of the interesting thing that they were able to do while, you know, some of the rest of the field that really tried to, tried to model different interactions separately, you know, how protein interacts with small molecules, how protein interacts with other proteins, how RNA or DNA, um, have their structure, they put everything together and, you know, train a very large models with a lot of advances, including kind of changing kind of some of the key architectural choices and managed to get a single model that was able to set this new state-of-the-art performance across, um, all of these different kind of modalities, whether that was protein-small molecules that is critical to developing kind of new drugs, uh, protein-protein, understanding, you know, interactions of, you know, proteins with RNA and DNAs and so on.

19:40 Just, uh, to satisfy the AI engineers in, in the audience [chuckles] what were some of the key architectural and data changes that made that possible? Yeah. So one critical one that was not necessarily just unique to AlphaFold 3, but there were actually a few other teams, including ours in the field that proposed this, was moving from, you know, modeling structure prediction as a regression problem, so where there is a single answer and you're trying to shoot for that answer, to a generative modeling problem where you have a posterior distribution of possible structures and you're trying to sample, uh, this distribution.

20:19 And this achieves two things. One is, uh, starts to allow us to try to model more dynamic systems. As we said, you know, some of these structures can actually take multiple structures. Uh, and so, you know, you can now, you know, model that, you know, through kind of modeling the entire distribution.

20:35 But on the second hand, from more kind of core modeling questions, when you move from a regression problem to a generative modeling problem, you are really tackling the way that you think about uncertainty in the model in a different way. So if you think about, you know, I'm undecided between different answers, what's gonna happen in a regression model is that, you know, I'm gonna try to make an average of those different kind of answers that I had in mind.

21:05 When you have a generative model, what you're gonna do is, you know, sample all these different answers and then maybe use separate models to analyze those different answers and pick out the best. So that was kind of one of the critical improvement. The other improvement is that they significantly simplified, to some extent, the architecture, especially of the final model that takes kind of those pairwise representations and turns them into an actual structure, and that's, uh, now looks a lot more like a more traditional transformer than, you know, like a very specialized equivariant architecture that it was, uh, in AlphaFold 3.

21:42 So this is a bitter lesson a little bit? There is some aspect of a bitter lesson, but the interesting thing is that it's very far from, you know, being like a simple transformer. This field is one of the, argue, very few fields in applied machine learning where we still have kind of architecture that are very specialized and, you know, there are many people that have tried to replace these architectures with, you know, simple transformers.

22:09 And, you know, there is a lot of debate in the field, but I think kind of the most of the consensus is that, you know, the performance that we get from the specialized architecture is vastly superior than what we get through a single transformer. Another interesting thing that I think on the staying on the modeling machine learning side, which I think is somewhat counterintuitive seeing some of the other kind of, uh, fields and applications, is that scaling hasn't really worked kind of the same, uh, in this field.

22:38 Now, you know, models like AlphaFold 2 and AlphaFold 3 are, you know, still very large models, but at the same time, they, in terms of parameters, they're actually not very big. They are definitely below a billion parameters. You know, if you hear these days in LLM space, you know, a model with less than a billion parameters, you'd think can't do anything.

23:00 But on the other hand, when you look at the computational cost of running these models, they're actually a lot more expensive than, uh, it is to run, uh, language models because as Jeremy was saying, we go from instead of having sort of like quadratic operations now a cubic operation. And so it's interesting how right now in the field, and, and this is maybe related to, you know, having kind of less data or, you know, needing more inductive biases, but we have this ratio of, you know, amount of computation to parameters that is much, much higher than in other, in other place.

23:36 If I recall, AlphaFold 2 was like, what, seventy million parameters? Something like that? Yeah. It's, it's something like that. It's quite... Yeah, it's quite small. It's less-- around a hundred or so. Yeah. These decisions of triangle layers and, like, these, uh, for AlphaFold 2, this, like, interesting equivariant architecture, like, really were priors that it baked in a lot of the physics of the system.

23:58 And also co-evolution data is, I think people have argued, that is kind of like almost like a database lookup of some sorts. Yeah. Yeah. It also sort of-- So that provides in some sense more parameters as well. Yeah. I mean, it's, uh, it's more definitely the amount of like, you know, pure like compute flops- Yeah...

24:15 right, is, is very high, and it's almost like more, yeah, more almost more like reasoning based maybe than like more just like information extraction. You know, I think one of the things that the, uh, part, part of the reason the LLMs are so large Isn't just because of their reasoning capability, but it's also because of like, like, the sheer quantity of information that they store.

24:32 And I think here there's a little bit less of that, you know, and I think it's more about like, you know, decoding this input rather than maybe like memorizing as much of it. So is there like a loop in the architecture that allows it to compute more for per parameter? Like, how does that work? Part of it is just, you know, exclusively this fact instead of, you know, having operations that operate on the single chain, they operate on the pairwise.

24:53 And so you instead of having like quadratic number of, of up kind of interactions, you have a cubic number of interactions. And so that on its own, you know, leads you to have, you know, smaller kind of representation sizes, but more representation that leads to more flops but fewer parameters. Right. Yeah.

25:10 On the other hand, you know, there is actually also this idea of, you know, they somewhat similar to, to reasoning where you recycle kind of this operation. So from AlphaFold2, but also kind of AlphaFold3, they have this interesting framework where, you know, you start... We-- as we were discussing kind of the input to the model is sort of like this initial understanding of the interactions either from the evolution of the multiple sequence, but also potentially from what we call templates that are basically database lookup of similar structures.

25:45 And so how the model works is that, you know, it decodes these and tries to understand a good, you know, potential raw structure of the pairwise interaction. And then what you can do is basically do this recycling where you feed this understanding back to the input of the model and then try to decode it again.

26:02 And people do this, you know, three or four times, and, you know, in some cases, you know, I've even tried to do it, uh, tens of times. And so you can see it as a very, very early version of kind of, uh, reasoning, uh, or, you know, trying to, uh, to get. Yeah. So you, you know, uh, AlphaFold2, really cool. AlphaFold3, really cool.

26:22 Um, but AlphaFold3 came with a catch, and I think this catch was important for the development of- Yeah... you know, Boltz and so on, so. Yeah. The catch was that it was an amazing paper, nature, uh, paper, but unfortunately, they, uh, decided not to release the model. Uh, you know, AlphaFold2, uh, was open source, and since then was, was used, I think the reported numbers is, you know, more than a million scientists.

26:49 AlphaFold3 for, you know, commercial reasons that, you know, um, DeepMind has since spin off as Amorphic Lab that is now trying to become sort of like a new pharmaceutical company, uh, had decided to keep this model internal and, and only use it internally. And now, uh, both, you know, we were in the field and, you know, building on top of models like AlphaFold, and so now we no longer had, you know, kind of the base starting point, uh, to build on top.

27:16 But even more importantly, everyone in both kind of academic research and in industry no longer had access to these incredible models that, you know, was, you know, really useful to try to understand biologists but also try to develop new therapeutics. I decided that, you know, to, to take the, the matter in our own hands and decided to kind of try to obtain a model that was of similar accuracy.

27:42 And so largely also, you know, using a lot of, you know, the, uh, information that was in the AlphaFold3 manuscript, we went ahead and built Boltz1, which was the first fully open source kind of model to approach the, the level of accuracy of, of AlphaFold3. And, you know, along the way and, and, you know, uh, we can talk about it more, but, you know, we realized that, you know, it was probably two ambitions to have, you know, to see this as, uh, an academic project.

28:11 And, you know, there were a lot of things that were kind of missing. And so we decided to also start a, a public benefit company to push kind of this, this mission of, you know, democratizing access to these models that we started with, uh, Boltz1. Quick interjection. I mean, I remember this. It was actually shocking how fast you got Boltz1 out. Like, it was just like two or three months, right? It was- I think we started in late May, and it came in November- Pretty long...

28:42 if I remember correctly. So slightly longer, but yeah. Yeah, it was relatively quick. I mean, for what it's worth, like, you know, we were working on some of the, some similar ideas at the time. I think like we, you know, for example, this idea of like having a diffusion model on top of, um, this like more, this pairwise strong was something that we were, we were exploring independently.

29:03 Now, when the paper came out, it was like really clear, like especially for example, on the data pipelines, there was like so much that we were like not really doing, and so there was a lot to like catch up on. But we were already in a place, I think, where we had, you know, so some experience working in, you know, with, with the data and working with these type of models, and I think that put us already in like a, a good place to, you know, to, to produce it quickly.

29:25 And, you know, and I would, [chuckles] I would even say like, I think we could have done it quicker. The problem was like for a while we didn't really have the compute. [laughs] And so we couldn't really train the model, and actually we only trained the big model once. Uh, that's how much compute we had. We could only train it once.

29:42 And so like while the model was training, we were like finding bugs left and right. Yeah. [laughs] Uh, a lot of them that I wrote. [laughs] And like I would-- I remember like I was like sort of like, you know, doing like surgery in the middle, like stopping the run, making the fix, like relaunching and yeah, we never actually went back to the start.

29:58 We just like kept training it with like the bug fixes along the way- Right. [laughs]... uh, which was- So it's impossible to reproduce now. [laughs] Yeah. Yeah, yeah. No, that model is like, has gone through such a curriculum that, you know, [laughs] it's learned some weird stuff. Uh, but uh, yeah, somehow by miracle it worked out.

30:13 The other, uh, funny thing is that the way that we were training, uh, most of that model was through, uh, a cluster from the Department of Energy. Yeah. But that's sort of like a shared cluster that many groups use. And so we were basically training the model for two days, and then it would go back to the queue and stay a week in the queue. Oh, man. [laughs] And so it was, it was, it was pretty painful.

30:35 And so we actually kind of towards the end with Devan, the CEO of, of Genesis, and basically, you know, was telling him a bit, a bit about the project and, you know, kind of telling him about this frustration with the compute. And so luckily, you know, he offered to kind of help and so we, uh, we got the help from Genesis to, you know, finish up the, the model. Otherwise, it probably would've taken a couple of extra weeks- Wow...

30:56 of which is- Yeah. Yeah. Boltz1 How did that compare to AlphaFold3 and then, and then there's some progression from there? Yeah. So I would say kind of the both Swan but also kind of these other kind of set of models that came, um, around the same time were kind of approaching were a big leap from, you know, kind of the previous kind of open source models and, you know, kind of, uh, really kind of approaching the level of AlphaFold3.

31:28 I would say-- still say that, you know, even to this day, there are, you know, some specific instances where AlphaFold3 works better. I think one, one common examples is antibody/antigen prediction where, you know, AlphaFold3 still seems to have an edge in, in many situations. Obviously, these are somewhat different models.

31:49 They are, you know, you run them, you obtain different results. So it's, it's not always the case that one model is better than the other, but kind of in aggregate we still, especially at the time, so AlphaFold3 is, you know, still having a bit of an edge. We should talk about this more when we talk about both Gen, but like how do you know one is-- one model is better than the other? Like you-- So you-- I make a prediction, you make a prediction. Like how do you know?

32:12 Yeah. So easily, you know, the, the great thing about kind of structure prediction and, you know, once we're gonna go into the design, uh, space of designing new small molecule, new proteins, this becomes a lot more complex. But a great thing about structure prediction is that a bit, uh, like, you know, CASP was doing basically the way that you can evaluate them is that, you know, you train, uh, the model on a structure that was, you know, released across the field up until a certain time.

32:37 And you know, one of the things that we didn't talk about that was really critical in all this development is the, uh, PDB, which is the Protein Data Bank, is this common resources, basic common database where every biologist can, uh, publishes their structures. And so we can, you know, train on, you know, all the structures that were put in the PDB until a certain date.

33:02 And then we basically look on recent structures, okay, which structures look pretty different from anything that was published before because we really want to try to understand generalization. And on this new structure, we evaluate all these different models. And so you just know when AlphaFold was-- 3 was trained, you know when you're-- you intentionally trained to the same date or something like that.

33:23 Exactly. Right. Yeah. And so this is kind of the way that you can somewhat easily kind of compare these models. Obviously, that assumes that you know the, the training set. You've always been very passionate about validation. I remember like DiffDoc and then there was like DiffDocL and DocGen. You've thought very carefully about this in the past.

33:41 Like actually I think DocGen is like a really funny story that I think, um, I don't know, I don't know if you want to talk about that. It's an interesting like- Yeah. I think one of the amazing things about putting things open source is that we get a ton of feedback from, from the field and, you know, sometimes we get kind of great feedback of people really liking the model.

34:03 [laughing] But honestly most of the times, um, you know, to be honest, that's also maybe the most useful feedback- Yeah... is you know, people sharing about where it doesn't work. And so, you know, at the end of the day it's critical and this is, you know, also something, you know, across other fields of, of machine learning.

34:20 It's always critical to set-- to do progress in machine learning is set clear benchmarks. And as you know, you start doing progress of certain benchmarks then, you know, you need to improve the benchmarks and make them harder and harder and this is kind of the progression of, you know, how the field operates.

34:38 And so, you know, the example of, of, uh, DocGen was, you know, we published this initial, uh, model called DiffDoc, um, in my first year of PhD, which was sort of like, you know, one of the early models to try to predict bio kind of interactions between proteins, small molecules, um, that we-- about a year after, uh, AlphaFold2 was published.

35:02 And now on the one end, you know, on these benchmarks that we were using at the time, uh, DiffDoc was, was doing, uh, really well kind of, you know, uh, outperforming kind of some of the traditional physics-based methods. But on the other hand, you know, when we started, you know, kind of giving these, uh, tools to kind of many, uh, biologists and, uh, one example was, uh, that we collaborated with was the group of Nick Palitzia at Harvard.

35:29 Uh, we noticed-- started noticing that there was this clear pattern where for proteins that were very different from the ones that we're trained on the models was, was struggling. And so, you know, that seemed clear that, you know, this is probably kind of where we should, you know, put our focus on.

35:48 And so we first developed, you know, with, uh, Nick and his group a new benchmark and then, you know, went after and said, "Okay, what can we change and kind of about the current architecture to improve this, uh, pattern generalization?" And this is the same that, you know, we're still doing today, you know, uh, kind of where does the model not work, you know, and then, you know, once we have that benchmark, you know, let's try to, uh, throw everything we, uh, any ideas that we have of the problem.

36:15 And there's a lot of like healthy skepticism in the field which I think, you know, is, is, is great and I think, um, it's very clear that there's a ton of things the models don't really work well on. But I think one thing that's probably, you know, undeniable is just like the pace of, pace of progress- Yeah... you know, and how, how much better we're getting, you know, every year.

36:32 And so I think if you, you know, if you assume you know any constant, you know, rate of progress moving forward I think things are gonna look pretty cool at some point in future. ChatGPT was only three years ago. [laughing] Yeah. I mean it's wild right? Like- What? Yeah, yeah. Yeah, it's one of those things like even being in the field you don't see it coming, you know, and like I think, yeah, hopefully we'll, you know, we'll, we'll continue to have as much progress we've had the past few years.

36:56 So this is maybe an, an aside but I, I'm really curious you Get this great feedback from the, from the community, right, by being open source. My question is partly, like, okay, yeah, if you open source, then everyone can copy what you did, but it's also maybe balancing priorities, right? Where you-- like all my customers are saying, "I want this."

37:20 There's all these problems with the model. Yeah, yeah, but my customers don't care, right? So like, how do you, how do you think about that? Yeah. So I would say a couple of things. One is, you know, part of, uh, our goal with Boltz and, you know, this is also kind of established as kind of the mission of the public benefit company that we started, is to democratize the access to these tools.

37:41 But one of the reason why we realized that Boltz needed to be a company, it couldn't just be an academic project, is that putting a model on GitHub is definitely not enough to get, you know, chemists and biologists, you know, across, you know, both academia, biotech and, and pharma to use your model to, uh, in their therapeutic programs.

38:03 And so a lot of what we think about, you know, at Boltz beyond kind of the just the models, is thinking about all the layers that come on top of the models to get, you know, from, you know, those models to something that can really enable scientists in the industry. And so that goes, you know, into building kind of the right kind of, uh, workflows that take in kind of, for example, the data and try to answer kind of directly that-- those problems that, you know, the chemists and the biologists are asking, and then also kind of building the infrastructure.

38:35 And so this to say that, you know, even with models fully open, you know, we see a ton of potential for, you know, products in the space. And the critical part about a product is that even, you know, for example, with an open source model, you know, running the model is not free. You know, as we were saying, these are pretty expensive model and especially, and maybe we'll get into this, you know, these days we're seeing kind of pretty dramatic inference time scaling of these models where- Yeah...

39:05 you know, the more you run them, the better the results are. Uh, but there, you know, you start getting into a point that compute and compute cost becomes a critical factor. And so putting a lot of work into building the right kind of infrastructure, building the optimizations and so on, really allows us to provide, you know, a much better service potentially to the open source models.

39:27 That to say, you know, even though, you know, we're the product, we can provide a much better service, I do still think, and we will continue to put a lot of our models open source, because the critical kind of role I think of open source models is, you know, helping kind of the community progress on the research and, you know, from which we, we all benefit.

39:47 And so, you know, we'll continue to, on the one hand, you know, put some of our kind of base models open source so that the field can, can build on top of it. And you know, as we discussed earlier, we learn a ton from, you know, the way that the field uses and builds on top of our models, but then, you know, try to build a product that gives the best experience possible to scientists so that, you know, like a chemist or a biologist doesn't need to, you know, spin off, uh, a GPU and, you know, set up, you know, our open source model in a particular way, but can just, you know, a bit like, you know, I even though I am a computer scientist, machine learning scientist, I don't necessarily, you know, take a open source LLM and try to kind of spin it off, but you know, I just maybe open the ChatGPT app or a cloud code and just use it as an amazing product.

40:37 Uh, we kind of want to give the same experience to scientists around the world. I heard a good analogy yesterday that a surgeon doesn't want the hospital to design a scalpel, right? [laughing] Right. Surgeon just buy the scalpel. [laughing] You wouldn't believe, like, the number of people, even like in my, uh, short time, you know, between, uh, a Fold3 coming out and, and the end of the PhD, like the number of people that would like reach out just for like us to like run off a Fold3 for them.

41:03 Oh, right. [laughing] You know, or things like that, just because like, you know, Boltz in our case, you know, just because it's like not that easy, you know, to do that com- you know, if you're not a computational person. And I think like part of the goal here is also that, you know, we continue to obviously build an interface with computational folks, but that the, you know, the models are also accessible to like a larger, broader audience and, and then that comes from like, you know, good interfaces and stuff like that.

41:28 I think one, like, really interesting thing about Boltz is that with the release of it, you didn't just release a model, but you created a community. Yeah. Did that community-- It grew very quickly. Did that surprise you? And like what is the, the evolution of that community, and how has that fed into Boltz- Uh, if you, if you look-... as Boltz company? If you look at its growth, it's- Yeah...

41:47 it's like very much like when we release a new model, it's like there's a big, big jump. But yeah, it's-- I, I mean, it's been great. You know, we have a Slack community, uh, that has like thousands of people on it. Um, and it's actually like self-sustaining now, which is like the really nice part because, you know, it's, it's almost overwhelming I think, you know, to be able to like answer everyone's questions and help 'cause it's really difficult, you know, with the, the few people that we were.

42:09 But it ended up that like, you know, people would answer each other's questions and like sort of like, you know, help one another and so the, the, the Slack, you know, has been like kind of, yeah, self, self-sustaining and that's been, that's been really cool to see. Um, and, um, you know, that's, that's for like the Slack part, but then also feel obviously on GitHub as well, we've had like a nice, nice community.

42:28 Um, you know, I think we also aspire to be even more active on it, you know, than we've been in the past six months, which has been like a bit challenging, you know, for us. But, um, yeah, the community has been, has been really great and, you know, there's a lot of papers also that have come out with like new evolutions on top of Boltz, and it surprised us to some degree because like there's a lot of models out there, and I think like, you know, sort of people converging on that was, was really cool.

42:53 And you know, I think it speaks also I think to the importance of like, you know, when, when you put code out, like to try to put a lot of emphasis in like making it like as easy to use as possible, and something we thought a lot about when we released the, the code base. You know, it's far from perfect, but, you know. Do you think that that was one of the factors that caused your community to grow, is just the focus on easy to use, make it accessible? I think so, yeah.

43:14 And we, we've, we've heard it from a few people- Okay. Yeah... over the, over the, over the years now, and, um, you know, and some people still think it should be a lot nicer. [laughing] And they're, and they're right. [laughing] Uh, and they're right. But, um, yeah, I think it was, you know, at the time maybe a little bit easier than, than other things.

43:29 The other thing part that I think led to, to the community and to some extent, I think, you know, like the, uh, somewhat the trust in the community and kind of what we, what we put out is the fact that, you know, it's not really been kind of, you know, one model, but-- and maybe we'll talk about it, you know.

43:45 After Boltz one, you know, there were maybe another couple of models kind of released, you know, uh, or open source kind of soon after. We kind of continued kind of that open source journey and released Boltz two, where we are not only improving kind of the structure prediction but also starting to do affinity predictions, understanding kind of the strength of the interactions between these different models, which is this critical component, critical property that you often want to optimize, uh, in discovery programs.

44:12 And then, you know, more recently, also kind of protein design, uh, model. And so we've sort of been building this suite of, of models that come together, interact with one another, where, you know, kind of there is almost an expectation that, you know, we, we take very at heart of, you know, always having kind of, you know, across kind of the entire suite of different tasks, the best or across the best model, uh, out there, so that it's sort of like our open source tool can be kind of the go-to model for everybody in the, in the industry.

44:47 I really want to talk about BoltzGen. But before that, one last question in this direction. Was there anything about the community which surprised you? Were there any like-- someone was doing something and you're like, "Why would you do that? That's crazy," or, "That's actually genius, and I never would have thought about that"? I mean, we've had many contributions.

45:05 I think like some of the interesting ones like-- I mean, we had, you know, this one individual who like wrote like a complex GPU kernel, you know, for part of the architecture. Oh. On a piece of-- The funny thing is like that piece of the architecture had been there since AlphaFold two. And I don't know why it took Boltz for this, you know, for this person to, you know, to decide to do it, but that was like a, a really great contribution.

45:32 We've had a bunch of others like, you know, people figuring out like ways to, you know, hack the model to do cyclic peptides. Like, you know, there's-- I don't know if there's any other interesting ones come to mind. One, one cool one, and, and this was, you know, something that initially was, uh, proposed as you know, as a message in the Slack channel by, by Tim O'Donnell was basically he was, you know-- There are some cases, especially for example, we discussed, you know, antibody-antigen interactions where the models don't necessarily kind of, uh, get the right answer.

46:02 What he noticed is that, you know, the models were somewhat stuck into predicting kind of the, the antibody to interact with the part of the antigen that was incorrect. And so he basically ran the experiments. In this model, you can condition, basically you can give hints. And so he basically gave, you know, random hints, uh, to the model, basically, "Okay, you should bind to this residue, uh, you should bind to the first residue, or you should bind to the eleventh residue, or you should bind to the twenty-first residue," you know, basically every ten residues scanning the entire antigen.

46:33 And- Residues are the, the- The amino acids... the amino acid, yeah. So the first amino acid, the eleventh amino acid, and so on. So it's sort of like doing a scan and then, you know, conditioning the model to predict all of them and then looking at the confidence of the model in each of those cases and taking the top.

46:48 And so it's sort of like a very somewhat crude way of doing kind of inference time search. But surprisingly, you know, for, for antibody-antigen prediction, it actually kind of helped quite a bit. And so there's some, you know, interesting ideas that, you know, as a, um-- obviously, as kind of developing the model, you, you say kind of, you know, "Wow, this is-- Why would the model, you know, be so dumb?"

47:10 But, you know, it's, it's very interesting and, and that, you know, leads you to also kind of, you know, start thinking about, okay, how-- why can I do this, you know- Yeah... not with this brute force, but, you know, in a, in a smarter way. And so we've also done a lot of work on that direction. And that speaks to like the, you know, the power of scoring. We're seeing that a lot.

47:29 I'm sure we'll talk about it more when we talk about BoltzGen. But, um, you know, our ability to like take a structure and determine that that structure is like good- Yeah... you know, like somewhat accurate, whether that's a single chain or like an interaction, is a really powerful way of improving, you know, the, the models.

47:48 Like sort of like, you know, if you can sample a ton and you assume that like, you know, if you sample enough, you're likely to have like, you know, the good structure- Mm-hmm... then it really just becomes a ranking problem. Yeah. And you know, now we're-- You know, part of the inference time scaling that Gabby was talking about is, is very much that.

48:03 It's like, you know, the more we sample, the more we like, you know-- the ranking model ends up finding something it really likes. Um, and so I think our ability to get better at ranking, I think is also what's gonna enable sort of the next, you know, next big, big breakthroughs. Interesting. I guess there's a-- my understanding, there's a diffusion model, and you generate some stuff, and then you-- I guess it's just what you said, right? Then you rank it using a score, and then- Yeah...

48:30 you finally, um-- And so, like, can you talk about those different parts? Yeah. So first of all, like the-- one of the critical kind of, you know, beliefs that we had, you know, also when we started working on Boltz one was sort of like the structure prediction models are somewhat, you know, our field version of some foundation models. You know, learning about kind of how proteins and other molecules interact, and then we can leverage that learning to do all sorts of other things.

48:58 And so with Boltz two, we leverage that learning to do affinity predictions. So understanding kind of, you know, if I give you this protein, these small molecules, how tightly is their interaction? Uh, for BoltzGen, what we did was taking kind of the kind of foundation models and then fine-tune it to predict kind of entire new proteins.

49:19 And so the way basically that that works is sort of like instead of for the protein that you're designing, instead of filling in an actual sequence, you fill in a set of blank tokens, and you train the models to, you know, predict both the structure of kind of that protein and with the structure also what the different amino acids of that proteins are.

49:42 And so basically, the way that, uh, BoltzGen operates is that you feed, uh, the str- A target, a protein that you may want to kind of bind to or, you know, another DNA, RNA, and then you feed the high-level kind of design specification of, you know, what you want your new protein, uh, to be. For example, it could be like an antibody with a particular framework, could be a peptide, could be many other things.

50:10 And that's with natural language or with this- And that's, you know, basically, you know, prompting and we have kind of this sort of like spec w- that you- Okay... you specify. And, you know, you feed kind of this, this spec to the model and then the model translate this into, you know, a set of, you know, tokens, a set of conditioning to the model, a set of, you know, blank tokens.

50:31 And then, you know, basically decodes as part of the diffusion models decodes a new structure and a new sequence, uh, for your, your protein. And, you know, basically, and then we take that and as Jerem was saying, you know, trying to score it and you know how good of, you know, a binder it is to that original target that you- You're using basically Boltz2 predict the folding and the affinity to that molecule.

50:59 So and then that is your-- that kinda gives you a score. Is that- Exactly. Yeah. So you, you use this model to predict the structure, and then you do two things. One is that you predict the structure- Mm-hmm... and with something like Boltz2, and then you basically compare that structure with what the model, uh, predicted, what Boltzen predicted, and this is sort of like in the field called consistency.

51:23 It's basically you want to make sure that, you know, the structure that you're predicting is actually what you're trying to design, and that gives you a much better confidence that, you know, that's a good design. And so that's the, the first filtering. And the second filtering that we did, uh, as part of kind of the, the Boltzen pipeline and that was released is that we look at the confidence that the model has in the structure.

51:49 Now, unfortunately, kind of going to your, your question of, you know, predicting affinity, unfortunately, confidence is not a very good predictor of affinity, and so one of the things that we've actually done a ton of progress, you know, since, uh, we released Boltzen and kind of we have some, uh, new results that we are gonna kind of announce soon is kind of, you know, the ability to get much better, uh, hit rates when instead of, you know, trying to rely on confidence of the model, we are actually directly trying to predict the affinity of that interaction.

52:23 Okay. Just backing up a minute. So your diffusion model actually predicts not only the protein sequence but also the folding of it. Exactly. Oh. And actually kind of the way, uh, one of the big different things that we did compared to other models in the space and, you know, there were some papers that had already kind of done this before, but we, uh, really scaled it up was, you know, basically somewhat merging kind of the structure prediction and the sequence prediction into almost the same task.

52:55 And so the way that Boltzen works is that you are basically the only thing that you're doing is predicting the structure. So the only sort of supervision is we give you a supervision on the structure. But because the structure is atomic and, you know, the different amino acids have a different atomic composition, basically from the way that you place the atoms, we also understand not only kind of the structure that you wanted but also the identity of the amino acid that, you know, the models believed was there.

53:26 And so we've basically instead of, you know, having these two supervision signals, you know, one discrete, one continuous that somewhat, you know, don't interact well together, we sort of like build kind of like an encoding of, you know, sequences in structures that allows us to basically use exactly the same supervision signal that we're using to Boltz2 that, you know, uh, you know, largely similar to what AlphaFold 3 pr- uh, proposed, which is very scalable, and we, we can use that to design new proteins.

53:58 Ah, interesting. Maybe a quick shout-out to Hannes, uh, Stark on our team who, like, did all this work. Um, yeah. Yeah. It w- that was a really cool idea. I mean, like, looking at the paper and there's this just like encoding where you just a- add a bunch of, I guess, kind of f- atoms which can be anything, and then they get sort of rearranged and then basically plopped on top of each other so that...

54:22 And then that encodes what the amino acid is, and there's sort of like a unique way of doing this. That was, that was, like, such a really, such a cool, fun idea. Yeah. I think that idea was-- had existed before. I think it wasn't- Yeah, there were a couple of papers- Yeah, yeah... that had, had proposed this and- Yeah... and Hannes really took it to, um, to the large scale. Yeah.

54:39 In the paper, a lot of the paper for BoltzGen is dedicated to actually the validation of the model. In my opinion, all the people we basically talk about feel that this sort of like in the wet lab or whatever the appropriate, you know, sort of val-- like in real world validation is the whole problem almo- or not the whole problem, but a big giant part of the problem.

55:00 So can you talk a little bit about the highlights from there that really... Because, uh, to, to me, uh, the results are impressive both from a, the perspective of the, you know, the model and also just the effort that went into the validation by a large team. First of all, I, I think I should tr- start saying is that both when we were at MIT and Thomas Yioskis and working at Barzilay's lab as well as at Boltz, uh, you know, we are not a, we're not a biolab and, you know, we are not a therapeutic company.

55:33 And so to some extent, you know, we were first forced to, you know, look outside of, you know, our group, our, um, team to do the experimental validation. One of the things that really Hannes and the team pioneered was the idea, okay, can we Go not only to, you know, maybe a specific group and, you know, trying to find a specific system and, you know, maybe overfit a bit to that system and, and trying to validate.

56:01 But how can we test these models across a very wide variety of different settings so that, you know, anyone in, uh, in the field and, you know, protein design is, you know, such a kind of wide task with all sorts of different applications from therapeutic to, you know, biosensors and many others that, you know...

56:22 So can we get a validation that is kind of goes across, uh, many different tasks? And so he basically put together, you know, I think it was something like, you know, twenty-five different, you know, academic and industry labs that committed to, you know, testing, uh, some of the designs from the model and some of this testing is, is still ongoing and, you know, giving, uh, results kind of, uh, back to us in exchange for, you know, hopefully getting some, you know, new great sequences for, uh, their task.

56:53 And, and he was able to, you know, coordinate this, you know, very wide, uh, set of, you know, uh, scientists. And, uh, already in the paper, I think we shared results from, I think, uh, eight to ten different labs, uh, kind of showing kind of results from, you know, designing peptides, uh, designing, uh, to target, you know, ordered proteins, peptides, uh, targeting disordered proteins.

57:19 We showed results, you know, of, uh, designing proteins that bind to small molecules. We showed results of, you know, designing nanobodies and across a wide variety of different targets. And so that sort of like gave to the paper a lot of, you know, validation, to the model, a lot of validation that was kind of, uh, wide, uh- And so those would be therapeutics for those animals or are they relevant to humans as well?

57:44 They're relevant to humans as well. Okay. Uh, obviously you need to do some work into quote-unquote humanizing them, making sure that, you know, they have the right characteristics to-- so they're not, uh, toxic to humans and so on. There are, um, some approved medicine in the market that are nanobodies. There's a general pattern, I think, in like, in trying to design things that are smaller, you know, like it's easier to manufacture.

58:08 At the same time, like that comes with like potentially other challenges, like maybe a little bit less selectivity than like if you have something that has like more hands, you know. Yeah. But the... Yeah, there's this big desire to, you know, try to, yeah, design mini-proteins, nanobodies, small peptides, you know, that just are just great drug modalities. Okay.

58:28 I think we were left off, we were talking about validation in the lab, and I was very excited about seeing like all the diverse validations that you've done. Can you- Yeah... go into some more detail about them? Yeah. The specific ones. Yeah. The nanobody one, I think we did, what was it? Fifteen targets? Is that correct? Fourteen. Fourteen targets. Testing.

58:52 So we-- typically, the way this works is like we, uh, make a lot of designs- Yeah... right? On the order of like tens of thousands. And then we like rank them and we pick like the top N. Uh, in this case, N was fifteen, right? For each target. And then we like measure sort of like the success rates, both on like how many targets we were able to get a binder for and then also in like more generally like out of all of the, you know, binders that we designed, how many actually proved to be, uh, good binders.

59:20 Some of the other ones I think involved like, yeah, like a-- we had a, a, a cool one where there was a small molecule and we designed a protein that, you know, binds to it. That has a lot of like interesting applications. You know, for example, like Gabby mentioned like biosensing and things like that. Okay. Uh, which is pretty cool. We had a disordered protein I think you mentioned also.

59:39 And yeah, I think some of-- maybe some-- those were some of the, the highlights. Yeah. So I would say that the way that we structure kind of some of those validations, uh, was on the one end, we have validations across a whole set of different problems that, you know, the biologists that we're working, uh, with came to us with.

1:00:00 So we are trying to, for example, in some of the experiments, design peptides that would, uh, target the RexC, which is a target that is involved in metabolism. And we had, you know, a number of other applications where we were trying to design, you know, peptides or other modalities against some other therapeutic relevant targets.

1:00:21 We designed some, um, proteins to bind small molecules. And then some of the other testing that we did was tr- really trying to get like a more broader sense of how does the model work, especially when tested, you know, on somewhat generalization. So one of the things that, you know, we, we found with the field was that a lot of the validation, especially outside of the validation that was done on specific problems, was done on targets that have a lot of, you know, known interactions in the training data.

1:00:53 And so it's a bit, always a bit hard to understand, you know, how much are these models really just regurgitating kind of what they've seen or trying to imitate- Yeah... what they've seen in the training data versus, you know, really be able to design, um, new proteins. And so one of the experiments that we did was to, uh, take nine targets from the PDB, filtering to things where there is no known interaction, uh, in, in the PDB. So basically, the model has never seen kind of this particular protein bound or a similar protein bound to another protein.

1:01:30 So there is no way that the model from its training set can sort of like say, "Okay, I'm just going to, uh, kind of-" Like tweak something. "... tweak something and, and just imitate this particular kind of interaction." And so we, we took these nine proteins, we worked with, uh, Adaptive, a CRO, and basically tested, you know, fifteen mini-proteins and fifteen nanobodies against each one of them.

1:01:55 And the very cool thing that we saw was that on two-thirds of those targets, we were able to, from these fifteen designs, get, uh, nanomolar, uh, binders. Nanomolar, roughly speaking, is just a measure of, you know, how strongly kind of the interaction is. And roughly speaking, kind of like a nanomolar binder is approximately the kind of binding strength of binding that you need for a therapeutic. Okay. Yeah.

1:02:22 So maybe switching, uh, directions a bit, Bolts Lab was just announced, um, this week. Or was it last week? Yeah. Yeah. This is, like, your first, I guess, product, if, if that's the- Yeah... if you wanna call it that. Can you talk about what Bolts Lab is and, um, yeah, you know, what you hope that people take away from this? Yeah.

1:02:44 You know, as we mentioned, like, I think at the very beginning, is the goal with the product has been to, you know, address what the models don't on their own, and there's largely sort of two categories there. I'll split it in three. The first one, it's one thing to predict, you know, a single interaction, for example, like a single structure.

1:03:06 It's another to, like, you know, very effectively search a space, a design space, to produce something of value. What we found, like sort of building of this product, is that there's a lot of steps involved, you know, in that. And we sort of need to, like, you know, accompany the user through. You know, one of those steps, for example, is, like, you know, the, the creation of the target itself.

1:03:26 You know, how do we make sure the model has, like, a good enough understanding of the target so we can, like, design something? And there's all sorts of tricks, you know, that you can do to improve, like, a particular, you know, structure prediction. And so that's sort of like, you know, the first stage.

1:03:38 And then there's, like, this stage of, like, you know, designing and searching the space efficiently. You know, for something like Bolts Gen, for example, like you, you know, you, you design many things and then you rank them. But for example, for small molecule, the process is a little bit more complicated.

1:03:52 We actually need to also make sure that the molecules are synthesizable. And so the way we do that is that, you know, we have a generative model that learns to use, like, appropriate building blocks such that, you know, it can design within a space that we know is, like, synthesizable. And so there's, like, you know, this whole pipeline really of different models involved in being able to design a molecule. And so that's been sort of like the first thing. We call them agents.

1:04:15 We have a protein agent, and we have a small molecule design agents. And that's really, like, at the core of, like, what powers, you know, the Bolts Lab platform. So these agents, are they, like, a language model wrapper, or they're just, like, your models and you're just calling them agents because they- Yeah... they, they sort of perform a function- Yes... on behalf of the user?

1:04:33 Okay. They're more of like a, you know, a recipe, if you wish. And I think, uh, we use that term sort of because of, you know, sort of the complex pipelining and automation, you know, that goes into, like, all this plumbing. So that's the first part of the product. The second part is the infrastructure. You know, we need to be able to do this at very large scale for any one, you know, group that's doing a design campaign.

1:04:55 Let's say you're designing, you know, I'd say 100,000 possible candidates, right, to find the good one. That is, you know, a very large amount of compute. Uh, you know, for a small molecule that's on the order of, like, a few seconds per design. For proteins it can be a bit longer. And so, you know, ideally you wanna do that in parallel, otherwise it's gonna take you weeks.

1:05:14 And so, you know, we've put a lot of effort into, like, you know, our ability to have a GPU fleet that allows any one user, you know, to be able to do this kind of, like, large parallel search. So you're amortizing the cost over your users basically. Exactly. Yeah. Exactly. And you know, to some degree, like, it's whether you use 10,000 GPUs for, like, you know, a, a minute is the same cost as using, you know, uh, one GPUs for God knows how long, right?

1:05:37 So you might as well try to parallelize if you can. So, you know, a lot of work has gone, has gone into that, making it very robust, you know, so that we can have, like, a lot of people on the platform doing that at the same time. And the third one is, is the interface, and the interface comes in, in two shapes.

1:05:52 One is in form of an API, and that's, you know, really suited for companies that wanna integrate, you know, these pipelines, these agents directly in existing, you know, workflows that they have, or, like, existing user interfaces that they have. And we're already, like, partnering with, you know, a few distributors, you know, that are gonna integrate our API. And then the second part is the user interface.

1:06:13 And, you know, we, we've put a lot of thoughts also into that. And this is when I, I mentioned earlier, you know, this idea of, like, broadening the audience. That's kind of what the, the user interface is about, and we've built a lot of interesting features in it, you know, for example, for collaboration.

1:06:26 You know, when you have, like, potentially multiple medicinal chemists are going through the results and trying to pick out, okay, like, what are the molecules that we're gonna go and test in the lab, it's powerful for them to be able to, you know, for example, each provide their own ranking and then do consensus building.

1:06:38 And so there's a lot of features around launching these large job, but also around, like, collaborating on analyzing the results, um, that we try to solve, you know, with, with that part of the platform. And so Bolts Lab is sort of a combination of these three objectives into, like, one, you know, sort of cohesive platform. Who is this accessible to? Everyone. You do need to request access today.

1:06:58 We're still, like, you know, sort of ramping up the usage, but anyone can request access. If you are an academic in particular, we, uh, you know, we provide, you know, a fair amount of free credit so you can, like, play with the platform. If you are a startup or a biotech, you may also, you know, reach out and we'll typically, like, actually hop on a call just to, like, understand what you're trying to do and also provide a lot of free credit to, to get started.

1:07:20 And, uh, of course, also with larger companies, we can deploy this, this platform in a more, like, secure environment. And so that's, like, more, like, custom, you know, deals that we make, you know, with, with the partners. You know, and that's sort of at the ethos of Bolts. I think this idea of, like, uh, servicing everyone and not necessarily, like, going after just, you know, the, the really large enterprises and that starts from the open source, but it's also, you know, main- ma- a key, key design principle of, of the product itself.

1:07:48 Yeah. One thing I was thinking about with regards to infrastructure, like in the LLM space, you know, the cost of a token has gone down by, I think, a factor of 1,000 or so- Yeah... over the last three years, right? Yeah. Yeah. And is it possible that, like, essentially you can exploit economies of scale and infrastructure that you can make it cheaper to run these things yourself than for any person to roll their own system?

1:08:07 Oh, 100%. Yeah. I mean, we're already there. You know, like, running Bolts on our platform, especially on, on a large screen, is, like, considerably cheaper, um, than it would probably take anyone to put the open source model out there and, and run it. And, and on top of the infrastructure, like, one of the thing that we've- Been working on is, is accelerating the models.

1:08:27 So, you know, our, our small molecule screening pipeline is ten x faster on Bolt's lab than it is in the open source, you know, and that's, that's also part of like, you know, building, building a, a, a product, you know- Yeah... something that, that scales really well. And we really wanted to get to a point where like, you know, we could keep prices, uh, very low in a way that it would be a no, no-brainer, you know, to, to use Bolts through, through our platform. Good.

1:08:52 How do you think about validation of your, like, agent tech systems? Yeah. Because, you know, as you were saying earlier, like we're-- AlphaFold style models are really good at, let's say, monomeric f- you know, proteins, where you have, you know- Yeah... co-evolution data. But now suddenly the whole point of this is to design something which doesn't have- Right... you know, co-evolution data, something which is really novel. So- Right...

1:09:15 now you're basically leaving the, the domain that you thought was- Yeah... you know, that you, you know you were good at. So, like, how do you validate that? Yeah. I like every complete, but there's, um, there's obviously, you know, a ton of computational metrics that we rely on, but those are only take you so far.

1:09:30 You really gotta go to the lab, you know, and, and test, you know, okay, with this method A and this method B, how much better are we? You know, how much better is my, uh, my hit rate? How stronger are my binders also? It's not just about hit rate, it's also about how, how good the binders are. And there's really like no way, no way around that.

1:09:49 And I think we're, you know, we've really ramped up the amount of experimental validation that we do so that we like really track progress, you know, as scientifically s-sound, you know, as, as possible. I don't know if there's anything. Yeah, no, I think, you know, one thing that is unique about us and maybe companies like us is that because we're not working on like maybe a couple of therapeutic pipelines where, you know, our validation would be focused on those.

1:10:15 We-- when we do an experimental validation, we try to test it across tens of targets. And so that on the one end, we can get a much more statistically significant result and, and really allows us to make progress from the methodological side without being, you know, steered by, you know, overfitting on any one particular system.

1:10:35 And of course, we choose, you know, we always try to choose, uh, targets and problems are sort of like at the frontier of what's possible today. So, you know, you don't want something too easy, you don't want something too hard, otherwise you're not gonna see progress. And so, you know, this is a somewhat evolving set of targets.

1:10:53 We talked earlier about the targets that we looked at with, with Bolgen, now we are even trying kind of, you know, even harder targets both for small molecule and proteins. And so we try to keep ourself on the, on the boundary, uh, of what's possible. So do you have like infrastructure or this is like you just have a lot of different partnerships with academic labs and you're just gonna keep pushing on these and driving these?

1:11:15 We do partially this through academic labs. Uh, more and more we do this through, uh, CROs just because of, you know, to some extent is also we need kind of replicability often kind of, you know, going after the same targets, you know, multiple times and, you know, to see the, the progress from, you know, one month to the next.

1:11:34 Um- And speed. And speed. Speed of execution. Yeah. And- So what happens if you start getting a bunch of like really strong binders against therapeutic targets? What do you do? Um, I mean- Release them. [laughing] Yeah. Put that, put the whole line- Oh, you release them in open source? Like- Yeah. I mean, you know, I mean, we're, we're-- when we say we have no interest in making drugs, we're serious. Like, you know, [chuckles] uh, I mean, when it- Yeah... when it was with the academic labs, basically the, you know, it was they keep it, they do whatever they want with it.

1:11:59 And with the CRO so far, yeah, we've been, we've been very, yeah, releasing, releasing them. I, you know, I, I will also say and, and I think this is a bit been a bit of the issue that I, I have with, with some of kind of the things that have been said in the field is when we say that we design new proteins or we say that we design new molecules, you know, go and bind these particular targets, we should be very clear, you know, these are not drugs.

1:12:25 You know- Yeah, yeah, yeah. [laughs]... these are not things that are ready to be put- Yes, yes. Yeah... into, into a human. And there is still a lot of development that, that goes with it. And so this is, this is kind of to, to us, you know, we see ourself as, you know, building tools for scientists.

1:12:41 You know, at the end of the day, you know, it really relies on the scientist having a great therapeutic hypothesis and then pushing through kind of all the stages of development. And, you know, we try to build tools that can accompany them, uh, in that journey. We-- it's not like a, a magic box where, you know, uh, you j-- can just turn it and get- Get FDA-approved drugs.

1:13:03 [laughing] FDA-approved drugs or FDA-approved drugs. Um, yeah. But actually that brings up an interesting question that I have, I've been wondering about is do you guys see yourself staying in this, for lack of a better way of saying it, layer? Or do you think that you'll start to either on a physical sense, looking at different layers of the virtual cell, so to speak, or also, you know, so there's the, like the development process that goes, you know, sort of like design, preclinical, clinical approval, and thinking about im-improving the performance throughout that process based on the designs.

1:13:43 Is, is that a direction that you guys are pushing? Yeah. So one of the things, as Jeremy said, you know, we are not a therapeutic company, and we want to kind of stay not to be a therapeutic company, always be at the service of, you know, all the different, you know, companies, including therapeutic companies that we serve.

1:14:01 And, you know, that to some extent does mean, you know, that we need to try to, you know, go deeper and deeper in getting these models better and better. One of the things that we are doing across, you know, uh, many other, uh, in the field is, you know, now that we are really-- they're start to be good both for small molecule and for proteins to design kind of binders, design relatively tight binders.

1:14:24 It's starting to look at all these other properties, you know, they call developabilities or ADME that, you know, we care about when developing a drug and trying can we design them from, uh, from the get go. The thing about those properties and some of them, you know- You need to, you know, s-start having an understanding of the cell.

1:14:45 And so that's on the one end kind of why we need that understanding, but also, you know, the way that we also think about all different and complex diseases is that these models and these tools that we're building have a good understanding of kind of, you know, biomolecular interactions and kind of their interactions.

1:15:01 Now, at the same time, every disease is often kind of unique, and every therapeutic hypothesis is unique, and so you maybe want to have something that needs to, uh, hit the particular, you know, let's say, target in a virus in a particular way, but you don't maybe know exactly what, uh, way you want to do.

1:15:21 And so maybe in the first set of designs, you're gonna try to target different epitopes in different ways, and then you're gonna test them in the lab, maybe directly in vivo, and you're gonna see which ones work and which one don't. And so then you need to bring those results back into the models, and then the models can start to have a more, uh, wider understanding, you know, not just of the biophysical of the, uh, antibodies interacting with that target, but also how that is shaped i- within the entire cell.

1:15:51 And so, first of all, you know, that means on the one end that we need, you know, kind of these loops, and this is also partially how we, we designed the platform to be, but that also means that we also need to start understanding more and more kind of higher level things. And, you know, I wouldn't say that we're working in any way on like a virtual cell like, uh, others are, but we're definitely thinking kind of very deeply about kind of, you know, how does, you know, kind of the way that we target, um, certain proteins interfere, interact with, you know, maybe pathways that are existing in the cell.

1:16:25 One question that has come up is you talk a lot about user interface and so on, and I think this is really important, but like my experience with dealing with medicinal chemists when you give them machine learning models is they are the most superstitious, skeptical, like pseudo-religious people I've ever talked to- [chuckles] -when it comes to doing science.

1:16:45 So- Sorry for the medicinal chemists listening. [laughs] Yeah. They're, they're amazing. Like, they're absolutely... So I've worked with some spectacular medicinal chemists who just pull magic out of their hat again and again, and I have no idea how they do it. But when you bring them a machine learning model, it is sometimes quite tricky to get them to deal with it.

1:17:03 How has your interaction been with this, and how have you thought about, like building Boltz Lab to work with the skeptics? One of the great value unlocks for us and for our product has been when we brought to the team a medicinal chemist. His name is, uh, Jeffrey. So I think kind of like on the one end, you know, day one, you know, he obviously had a lot of opinions on kind of a lot of the ways that we should, uh, uh, change, you know, both kind of the way that the agents worked, the way that the platform worked.

1:17:31 Uh, but it's been really amazing kind of, you know, once also we started kind of shaping kind of the platform, uh, in a better way with, with his feedback, how we went from, you know, some extent, you know, a fair skepticism to him, you know, actually using a lot more compute than any of our- [laughs] -computational, uh, folks in the team.

1:17:54 You know, um, at times that, you know, he's, you know, running, you know... He has all these sort of ter-- uh, hypotheses. "Okay, maybe I can hit this protein this particular way. I can hit it in that way. Actually, let me look at for this particular molecular space. Uh, let me try to optimize for these particular interactions."

1:18:13 So he ends up, you know, running several screens in parallel, you know, using hundreds of GPUs, you know, on his own. And, you know, so this has been, you know, pretty incredible to see kind of how, you know, maybe the way that I was more thinking about a problem, which is, okay, you're just trying to design a binder, a small molecule to a particular protein.

1:18:31 The way that he thinks about it is, you know, much more deeply and, you know, trying all these different things, these different hypotheses. And then, you know, once he gets the results from the model, he doesn't just, you know, take the top fifteen, uh, but he, he really kind of looks over and, you know, kind of tries to understand, you know, the different things.

1:18:48 And then when we, uh, select, you know, maybe some designs to bring forth, you know, he has, you know, something where, you know, both the models understand that something's good, but all-- himself as well. And that's why we also built kind of the platform to be, uh, an interface for, you know, this kind of, uh, this kind of chemist and, you know, also like a collaborative experience.

1:19:09 I, I think at the end of the day, like, you know, for people to be convinced, you have to show them something that they didn't think was possible. And until you have that aha moment, you know, I think the skepticism will remain, but then when, you know, every once in a while I think there's like a, a result that like really surprises people and then it's like, "Oh, wow, okay, this actually-- I can do something with this."

1:19:28 So you just get it in their hands, have them try it out, and they'll be convinced. Yeah. Yeah. Or like at maybe once- Some of them... the lab results come back. [laughs] Or their, their friend... Yeah, or maybe one of their colleagues is convinced and- Yeah. Yeah. I think it, it, it takes going to the lab- Yeah... I think at some point. There's no avoiding that.

1:19:44 You know, as beautiful as the platform can be, as nice as the molecules might look, you know, that the model predicted, I think what really convinces people is like, you know, hits. Yeah. Yeah. You see the results and yeah. Exactly. Yeah. Cool. Thank you for, you know, taking the time to chat with us. Yeah. It's been really interesting. Um, you know, is there anything that you would like your audience to know?

1:20:05 I mean, first of all, you know, uh, we're just getting started, you know, uh, continuing to, to build the team, and so, um, definitely always looking for, uh, great folks both on, you know, kind of, you know, software side, you know, machine learning side, but also scientists, uh, to join the team and help us, you know, uh, shape- On the infrastructure side too?

1:20:27 Like- Indeed. Um- If you, if you think that if you want a new challenge, because this is not just next token prediction, this is really a new engineering challenge- Exactly... that hasn't been done before. If you-- If-- No matter, you know, how much experience you have- Mm-hmm... with, you know, biologists and chemistry, if you want to come, you know, help us, you know, shape what, you know, biology and chemistry hopefully will look like in five, ten years, um, we'd love to hear from you.

1:20:52 And so, um, go to Boltz.bio and, you know, come join the team. Cool. Thank you. Awesome. Yeah. Thank you so much. Thank you so much. Thanks so much. Thank you. [outro jingle]