Mistral: Voxtral TTS, Forge, Leanstral, & what's next for Mistral 4

[on-hold music] Okay, welcome to Latent Space. We're here in the studio with trusty co-host, Vibhu. Welcome. [chuckles] Very excited for this one. As well as Guillaume and Pavan from Mistral. Welcome. Excited to be here. Thank you for having us. Pavan, you are leading audio research at Mistral, and Guillaume, you're our chief scientist. What are we announcing today? We-we're coordinating this release with you guys. Yeah.

So we are releasing Voxtral TTS. So it's our first audio model that generates speech. It's not our first audio model. We had a couple of releases before. We had one in the summer that was Voxtral, our first audio model, but it's-- it was like a transcription model, so ASR. Like a few months later, we released some update on top of this, supporting more languages.

Also, a lot of table stake features for our customers, context biasing, position, timestamping on the transcription. We also had some real-time model that can transcribe not just at the end of the... It was-- d-don't need to fill your entire audio file, but that can also come in real time. And here this is a natural extension in the audio, so basically speech generation. So yeah.

So we support nine languages, and this is a pretty small model, 3D model, so very fast and also startups, yeah. The performance is the same level of the, of the best model, but it's much more efficient in terms of cost and also much... In terms of cost, it's also much cheaper. Only a fraction of the cost of our competitors.

And we are also releasing the weight of this model around the interface. Yeah. May I be linked? Not just that. Well, yeah. What's the decision factor? [chuckles] It's a good question. There'll be more. There'll be more. [chuckles] Ooh. Yeah. Pavan, any other sort of research notes to add on what- Maybe we'll dive into it later in the forecast too, but it's a novel architecture that we developed in-house.

We iterated on several internal architectures and ended up with a autoregressive flow matching architecture and also have a new in-house neural audio codec, which converts this audio into 1.5 herds latent tokens, semantic and acoustic tokens. And yeah, that's, that's the new part about this model, and we're pretty excited that it, it came out with such good quality. And Guillaume was mentioning, yeah, it's a 3B model.

It's based off of the Mistral model that we actually released just a few months back and inside Trunk. And it mainly meant for like the TTS stuff, but the need text capabilities are also there. Yeah. So there's a lot to cover. I always-- I love any-anything to do with novel encodings and all those things because I think that's obviously increased a lot of efficiency, but also maybe bugs [chuckles] that sometimes happen. You were previously at Gemini, and you worked on post-training for language models, and maybe a lot of people will have less experience with audio models just in general compared to pure language.

What did you find that you have to revisit from scratch as you joined Mistral and started doing this? At least when it comes to-- for-- I think there are, there are two buckets, I guess. The audio understanding and audio generation. The audio understanding, like the Voxral models that Guillaume was mentioning that we released earlier, the Voxral chat that we released ear- I think July last year, and the follow-up transcription-only models family that we released in January.

That would be one bucket, and the generation is another bucket. I think you can also treat them as a unified set of models, but currently the approaches are a little different between these two to your question on how audio is fed to the model. In the understanding model, it's very similar to actually Pixral models that we also released- Yes, that was amazing... a few months back. Yeah.

[chuckles] It was pretty-- I-- That was the first project I worked on after joining Mistral. It was pretty, pretty nice. And Voxral was very similar in spirit, I guess. So we feed the audio through an audio encoder similar to images through a vision encoder, and it produces continuous embeddings and which are fed as tokens to the main transformer-- decoder transformer model. Yeah.

And the model output is just text. So on the output side, there is nothing that needs to be done in these kinds of models. I guess the interesting part about the generation step is the output now has to produce audio. And the approach that we have is this neural audio codec, which converts audio into these latent tokens.

There is a lot of existing literature and a lot of models which are based off of this kind of approach, and we took a slightly, uh, different design decisions around this. But at the end of the day, the neural audio product converts audio into a 12.5 hertz set of latents, and each latent is-- has a semantic token and a set of acoustic tokens.

And the idea is that you take these discrete tokens and then feed it. On the input side, there are several ways to fuse this at each frame, but we just sum the embeddings. So it's like having key different vocabularies and combine all of them because they all correspond to one audio frame on the input side. The output side is the interesting part.

On the output side, the-- It's not the-- I don't know if it's the most popular, but one popular technique is to have a depth transformer because you have key tokens at each time step. Like with our text, you just have one token at each time step. So you just do predict the token from the vocabulary with, yeah, with just, you get probabilities- This is a very straightforward text. Very straightforward.

Yeah. But if you have key tokens, then the main thing would be to predict all of them in parallel, but that doesn't work. At least that doesn't work that well because audio has more entropy. And the-- One of the techniques people use is this depth transformer, where you, you almost have a small transformer, or it can be LSTM-RN as well, but people use transformers, and you predict the key tokens in autoregressive fashion in that.

So you have two autoregressive things going on. So the thing we did differently is instead of having this autoregressive key step prediction, we have a flow matching model. Instead of modeling this as a discrete token set, we train the codec to be both discrete and continuous to have this flexibility.

So we did try the discrete stuff too, and which it works well, but the continuous stuff works just better. So yeah, we took this flow matching. So the-- it's a flow matching head which takes the latent from the main transformer- And like, kind in diffusion, it's denoising, but in this flow matching, it's a velocity estimate.

So you go from this noised all the way to the audio latent, which corresponds to the eighty-millisecond audio, and then which is sent through the vocoder to get back the eighty-millisecond audio frame. Yeah. Is this the first application of flow matching in audio? 'Cause usually I come across this in the image- Yeah.

Actually, it-- in some sense, uh, th-there are models, flow matching models in audio, but I think this specific combination- Mm. I could be wrong. There could be someone. [chuckles] No, no. I haven't seen, I haven't seen much work in this, so I, I think it's novel. And a lot of... it's just a way bigger community.

So they-- I think they pioneer a lot of these diffusion flow matching work, and it's interesting to adopt some of the ideas there into audio and- Yeah. Yeah. And personally, that's the thick part which is trying out. One of-- And more meta point is, unlike text, even in vision, I think this is true, but in audio it's definitely true, is that there is no winner model yet. There is no, "Okay, this is the way you do things."

It's, it's still evolving. I think people are still iterating and figuring out like, what's the best overall recipe, I guess. The idea, pretty sure there are models which are also completely end-to-end, like native audio and native audio, but it's still not come to a convergence point where this is the right way to do.

And that, that also makes the, uh, space pretty exciting to explore. What are some of the ways to look at it? There are ways where you can do diffusion for audio generation, but if you want, like, real-time generation, that's a big thing with the approach I'm assuming that you took. Yeah. And also, like, how do you go about evaluating different axes of what you care about? Yeah. Good point. I think we-- So you can do just flow matching diffusion for the whole audio.

We didn't even go down that path because one of the main applications is voice agents, and we want real-time streaming, and that's the use case. That's not the only use case, but that's one of the primary use cases we want to get to. So we picked the autoregressive approach for that. And within the autoregressive space, again, you can do chunk by chunk or you can do... So we picked the-- I think at least personally prefer the approaches which are the simplest.

And so we tried to see can we just add audio as just another head to our regular de- transformer decoder model because that kind of makes it easier for eventual end-to-end modeling of audio text native modeling. Yeah, and it works pretty well. So I guess we went with that. And we tried it a little bit with the flow matching head itself, like we had a discrete diffusion kind of approach, which also works well, but the flow matching worked better.

I was just curious about how you also think about this overall direction of research. Do you-- Basically, when you work with the audio team, do you set some high-level parameters and then let them explore whatever? Or how does it work between you guys? No, I think the way it works is that we have, uh, we are prioritizing together, I think what are the most important features, because there are many things we can do in audio. Yeah.

I think we try to decide like how we should do things. For instance, ultimately, what we want to do is to build this full duplex model, but we are not going to start this, start there directly. I think it's some of my project people are doing. But- Just to confirm, full duplex means it can speak while I'm speaking or- Yeah. Okay. Hear audio in you what to say. Yeah, yeah.

So ultimately, we are going to get there. But for us it was we decided to take it like step-by-step. So we start with whatever is the most important, I think also for our customers, which is a transcription, is the most popular use case. Then the speech generation was a real time just a bit before that.

And then the next thing is going to be like more like try combine everything all together. But, but yeah, we thought it was also important to like separate things and optimize each capability one by one before we- Mm. merge all of that together. Then the super omni model, um- But very interesting because as Pavan said, it's when you work on some other domains of this LLM and everything, there are many areas where I think it's not as interesting.

For instance, many places it's essentially just around data or like creating new environments and a lot of kind of easy things, but things where I think the research is maybe not as interesting where in audio there are so many ways to actually build these models, so many ways to go around it. That is-- The science I think is really interesting. And what we also tried for speech generation is that we tried multiple approaches.

What was interesting is that even though they were extremely different, they ended up big at the end of the day, like the particulars, but the flow matching turned out to be quite more natural. So we are happy with this. Is there an intuition why? It maybe like flow matching is just models speech better in some natural fundamental latent dimension?

[chuckles] No, I think the main thing is e-even at a particular time step, there is a distribution of things- Yes... to be predicted, like the way you inflect. So you already know the word that you're speaking, and you have the in text space, let's say the word maps to just a single token for simplicity. In most cases it does.

So there is not a lot of-- So you just pick the word. But with-within audio, even the same word could, even with your own voice, could be inflected in so many different ways. And I think a-any approach which like models this distribution and, and flow matching is one, one of the tech-- It-it's not the only one at all, but it's, uh, one which works pretty reasonably well, I think that's better.

So you have to pick across several different... The intuition I have is it's-- there are some several different clusters, each corresponding to some specific way you would inflect pronounce that thing, and you can't predict the mean of it because that corresponds to some blurred out speech or something like that.

But you have to pick one and then like sharp- Conditional inference. Yeah, exactly. Is that all covered under disfluencies, which is I think the normal term of art? Uh- Disfluencies, pauses, intonations. By the way, I, we have to thank Sophia for setting all this up, including like some of these really good notes because- Yeah... I'm less familiar with the audio domain.

[chuckles] I think disfluencies are definitely one such phenomenon. Uh, disfluencies is more like- Which is ums, uhs. Yeah. Um, uh, and also repeat like you feel like- Yeah, yeah... you do these filler words, you're thinking, so you repeat the word. Okay. And- Whereas intonation is like a dif-- it's up, up speak- Yeah... and all this. Okay. And yeah.

So I think there is a lot of like entropy and modeling it as a distribution and a-any technique which helps with it. And the depth transformer is a conditional way of modeling this, and transformers are actually really good at it, even though that's a mini transformer. So I think that worked pretty well too for us too.

It's just that the main consideration is when you have a depth transformer, if you have K tokens, you need to do K autoregressive steps. Even though it's a small thing, it's K steps, which is very latency heavy. Mm. With flow matching, we were able to cut it down significantly, so we are able to do the inference in quad steps or sixteen steps, and it works pretty well.

And there are more normal techniques to bring it down even further to like in the extreme case, one step. Like we're not doing it yet, but at least the framework lends itself to more efficient and- Yes. And the image guys have done- Yeah. Like incredible work as, uh- It-- now you just send the prompt and you get an image. Yeah. Surprisingly not enough, I think image model labs use those techniques in production.

I think it's-- I feel like it's a lot of research demos, but nothing I can use on my phone today. The thing is, there's a thing that would be interesting here is that since indeed there has been so much work that has been done in the vision community compared to audio on this domain. I think there are so many learning flows here. Yeah. And there are so many things we can do to actually improve this by even further.

So I know it's our first version, but we have so many ways to make this much better and much more efficient, cost efficient. So yeah. Yeah. So really it's not a new field at all, of course, but there are still so many things that can be done for it. I should also mention for those who are newer to flow matching, I think the creator is-- this guy's name is Alex. He's done, I think in Europe maybe two years ago. There was a very good workshop.

There's one hour on like this, what flow matching is. I would recommend people look that up. That's the other thing, right? The efficiency-wise, like I, I imagine like the reason it's open weights, the reason you pick three point six B backbone, it-- or three point four B, you are trying to fit to some kind of hardware constraints.

You're trying to f-fit some kind of like basic constraints. What are they? Not necessarily. Yeah. I think something we care about in our model is they are efficient. So we have a lot of separate model, for instance. So we have this audio model that is very small, very efficient. We also have a small OCR model that is really very good, very efficient as well.

And I think on that project, maybe other, I think companies are going to take is to have like a very general model that will do a bit of everything, but that is also going to be expensive. And here what you want to say is, if you care about this specific use case, if you can actually use this model, it just does that. It's extremely good at it, but also very efficient.

That's why we can actually add these models, audio, but also OCR that are like really good at that and that will be much more cost effective than those, than a general model that will contain a lot of capabilities you don't really need at the time. So yeah, so we are doing like general model, but also like more customized model like this. How does it compare to other TTS models? It's we're going full open weight. We're just dropping it. I think it's pretty good.

Yeah, I think it's pretty good. Like it, it's definitely one of the best, for sure. It's probably-- I, I would say it's the best open source model wise. You can really decide for themselves. Yeah. Why now? How does it fit into broader Mistral vision? How do you see voice agents? How do you see voice? I think every year I've heard, "Okay, you're a voice. You're a voice." There's a lot of architectural stuff.

There's a lot of end-to-end latency that you're solving, but where do you see voice heading? We had so many customers asking for voice. That's also why we wanted to build it. What's interesting in this domain is that in a sense, if you take something simple like transcription, it doesn't seem like something that should be very hard to do for a model.

Essentially, it's pattern recognition, it's classification, and these models are very good at classifying, right? And nonetheless, when you talk to them, it's not there yet, right? It's not-- you don't talk to them the same way you talk to a person or something. Maybe people don't realize it. In English, it's still much better than in any other language.

Even compared to French, for instance, if you talk to this model in French, when you see people talking to this model, they will talk very slow. They will articulate as much as they can. [laughs] So it's not natural, right? We are not yet to this. And I think, yeah, maybe the next generation will not know this.

But yeah, I think people that are maybe our age will actually always keep this bias of speaking very slowly when they talk to this model, even if maybe probably in a couple of years or maybe next year, it will not be necessary anymore. But yeah. But what's interesting is to see that, yeah, even for like languages like, yeah, French and Spanish, German that are not low res-low resource languages, you have a lot of audio just there and still it's not as good.

And I think the conse-- I mean, the reason for this, I suppose, is just there is not as much energy, as much effort that has been put in some other modalities like, for instance, vision or like coding. But, but yeah, there is still a lot of progress to be done. I think it's just a question of doing the work, and it's like a clear path of things to get there.

It's a little fascinating because I worked on Google Assistant, I think a while back at this point, but [chuckles] it's, I think it's like when you take a step back, it's fascinating. It's not that long ago. It was like four years ago or five years ago, and it's now it's completely audio in, audio out, and the function calling and the whole thing happens completely end to end and in a very natural- Yeah...

natural way and still ways to go. You were saying even despite all the progress, it's not like you're speaking to a person when you talk to any of these agent bots or voice mode kind of situation. It's still like a gap. I think that's the great part, and I feel like with even the existing stack, we should be able to get to this very natural speech conversational abilities soon enough, I guess, and we'll also hope, hope to get there.

And it's kind of the next step, right? Because when you talk to these agents, like usually people are just writing to them or sometimes there is this very clear, for instance, you are-- you want to write code, but you are-- you have a very clear idea of how you want the model to implement what you have in mind.

But so here you are going to spend like a lot of time writing, so it's not really efficient. And audio is really like a natural interface that is just not there yet, but I think it's just going to be there very soon. How is it like building, serving, inferencing? Like we see a lot about it's very easy to take LLMs off the shelf, serve them, fine-tuning, deploying.

I know you guys have a whole-- you have Forge, you have a whole stack of customizing, deploying. Is there a lag in getting that like distribution channel? Are you hoping there is-- so like prompting LLMs, you can have them be concise, verbose, all that. They're built on LLM backbones, these models. How do you see all that?

Yeah, I think this is a lot of what we are doing with our own customers. Very often they come to us, so it's for different reasons. I think one reason is sometimes they have this lot of privacy concerns. They have this data that it is very sensitive. They don't want the data to leave the company. They want it to stay inside the company.

So we help them deploy model in-house, so either on a-- either on premise or on private cloud. So they are not worried that it's given to a third party and that there is some leakage. Sometimes they have this kind of... Many, many companies have this different sensitivity of data. They have like sometimes tier one, tier two, tier three data. Tier three can send it to the cloud. Tier one it has to stay there.

So then it creates some kind of heterogeneous workflows where it's annoying and you cannot send some data to the cloud. This one you can. So here when we actually deploy the model for them is they don't have this consideration. They are like not worried that this is going to leak. Everyth-everything is much easier, so we help them basically do this on the...

So it's one of the value proposition, but, but the other is very often when customers use this off-the-shelf closed model, what's very sad is that they are not leveraging this data that they have been collecting for, for years or sometimes for decades. So much data, sometimes it's trillions of tokens of data in a very specific domain, their domain, which is data that you will not find on, in the public, uh, on the public internet.

So data on which, like the closed model, we actually not have access to one which is going to be really good. So if they are using like closed-source models, they are basically not benefiting from all these insights, all this data they have collected through years. They can always give it into the context that it finds, but it's never as good as if you actually trained the model as is. So, so yes, that's basically what we help them to do.

We actually provide them some Mistral approach is basically what we announced at, uh, GTC this week. So we provide them with this. It's basically like a platform with a lot of tools to actually help them process data, train on that. Yeah, it's actually the same thing that we are using in the science team.

So it's actually very battle-tested infrastructure, like a lot of efficient training code base for continued pre-training, like a fine-tuning, even doing SFT, RL. So we help them do this using the same tools as what our science team is building, is using. So since it's tools that we have been using for two years now, it's really battle-tested, it's really sophisticated, so it's the same thing we are giving to them.

We are giving the company the same thing that what our science team using internally to actually build their own AI. And it makes a very big difference, I think. Sometimes customers, um, many people in general, don't realize how much better the model becomes when you fine-tune it on your own data. And you can have your model is here, and you start from there, and you have a closed source model, which is sort of here. But if you actually fine-tune, it can actually really, really go much further than this.

And then you have a very big advantage. The model is trained on your entire company knowledge, so it knows everything. You don't have to feed like ten K tokens of context at every query. So it's, it's much easier. It's a bit... I think using a closed source model is really sad because it basically puts...

You are not leveraging all this data, and you are going to be using the same model as all your old competitors when you could actually using everything you have been collecting for years, which is really valuable. So yeah, so we help basically customers do this. We have a lot of solution being deployed for the engineer that go in the company that basically look at the problem customers are facing.

They look at what they are struggling to do, what we should do to solve it, so we help them solve them together. So it's, I think our approach is a bit different than theirs and some other companies and competitors. It's, we don't just create an endpoint and say, "Do some stuff on top of that," or we don't just give a checkpoint. We, we really look very closely with customers. We look at the issues they have.

We help them solve them. We really make some tailored solution for the problems they are facing. Some example are also going to be, sometime we have some customers, they really wanted to have a really good model, really performant on some like Asian rare languages. On the... If you take some off-the-shelf models, they, they can speak it. They can write in this language, but it's not amazing.

This language will be like maybe zero or one percent of the mixture, so it has been included during training, but very little. So what we did here is, of course, we trained a new model for them, but so this language was fifty percent of the mix, so it's much, much stronger. It knows of the dialects, it knows slang. So it's...

Yeah, so it's some example of things we can do. And it's really arbitrarily custom, I think. You have some of their customers, for instance, they wanted some, um, they wanted some 3D model that can do audio with a very good at function chain. So something you wanted to put in the car. In particular, they wanted this to be offline because in a car you don't necessarily have access to internet. So yeah, so here we can actually build these solutions.

There is no, like, model out of the box on this in the internet. You have this very, you have this very general model, a generalist, like, reasoning and strong model. But for things like this, they, they always want, like, specific solutions, uh, on here. And some of the reasons sometimes they come to us is because, like, they, they experiment with some closed source model, or they get some prototype. They are happy with what they build. They... It works well.

They're happy with the performance, and then they want to go to production, and then they realize, oh, but it's extremely expensive. We cannot push this itself. So then they come back to us and they say, "Can you help, help us build the same thing as this, but using something much cheaper on here?" On here, we can sometimes build something ten X cheaper by just fine-tuning a model, or it would be better on-prem, uh, on their own server, and also much cheaper as well, so yeah. That's the Mistral pitch right there.

[laughing] Take all the money. And, and I mean, outside of that, you do, we do put open weight models, so people can do this themselves. I feel like not enough people go out of their way to- They're not going to. They're gonna ask us to do it. [laughing] Ask the expert. Initially, we didn't know. We were not competition at the beginning of the company because I think our strategy was not exactly the same as what it is today.

But what we underestimated initially is the complexity of deploying this model and connecting them to everything to be sure it has access to the company knowledge. On the... And it was, yeah, on... We were seeing customers struggling with this, but it was even... That was two years ago, and now things are much more complicated because now you don't just have text on SFT, on simple instruction following.

Now you have reasoning, like agents, you have like, uh, tools, and you have multimodal audio. So it's much more complicated than before, and even back then it was hard for customers, so they really need some support, and this is why we're actually providing, like, always some for deposition as well as deploy this.

I'm, I'm curious, is there also voice fine-tuning that people do? So in this forge, we also have a- Oh, really?... a uni-unified framework, and the hope is like the Voxer's speech-to-text that we released earlier this year, and even the Voxer chat that we released last year. And I think a big... People...

I think there's a big, rich ecosystem of people fine-tuning Whisper, and people want the same thing with Voxer. It's much stronger than Whisper, and yeah, the, the platform offers that kind of fine-tuning. Yeah. Which could be any kind of fine-tuning. Like for instance, even sometimes people want to support new languages to this, which are tail languages, which we hope to cover our set natively.

But if there is a language where you have data and you want to fine-tune, I think this is a good use case. Or the other use case is you... It's the same language, like even English, but it's in a very domain-specific way where- Yeah, terminology, jargon, medical stuff. Exactly. Yeah. And also the specific acoustic conditions, like there's a lot of noise- Noise... or the...

And the model will do decently in most conditions, but you can always make it better, and the, those are some of the use cases where you can improve it e-even further, and that's one good use case for this. And for text-to-speech, we're just releasing it, so we'll have support for that soon, too. I think it's similar use case.

It's little different, the kind of things that you want to extend a text-to-speech model to, which could be like voice personalization, voice adaptation for enterprises and Many enterprises need very specific kind of tone, very specific kind of like personality for this kind of voice, and all of those are like good use cases for fine-tuning.

This I was gonna ask you. We never talked about cloning, voice cloning here. How important is it, right? Like I just clone a famous person's voice. Okay, but... Well, the main use case would be like for enterprise personalization. Like enterprises need like lot of customization. You don't want the same voice for all the enterprises.

Each enterprise want a customized, specialized, something which is representative of both their brand and also their, I guess, safety considerations. And the use case, I think the kind of thing that you would deploy as a empathetic assistant in the context of a healthcare domain would be very different from the kind of thing that would be in a customer support bot and would be different from like more conversational aspects.

I think tho- those are the customizations you would expect from enterprise, and that's the main use case, at least from our side. My, my basic example is you don't wanna call two customer services and have the same exact voice. [laughs] It's just, it's gonna be weird. But also on the technical side of this, so there's like a few things in Voxtro that I thought were pretty interesting. He's a big fan of this paper. [laughs] He said- Very good paper.

I think he said this is the best ASR paper he's ever read. Yeah, yeah. I've hyped up this voice paper enough. We covered it somewhere. But a big thing, so Whisper is known for thirty-second generation, thirty-second processing. You extended this to forty minutes. There was a lot of good detail in the paper about how this was done, even little niches of how the padding is so... It's very much needed. You need to have that padding in there.

The synthetic data generation around this, I'm wondering if you can share the same about the new speech to text, right? Text to speech. So how do you, how do you generate long form coherent? How do you generate... How do you do that? And then any gems. Is there gonna be a paper? Yeah, yeah. There would be a technical report. Okay.

Yeah, I think it'll have a lot of details, but m- I think the summary of it, actually some of the considerations in this paper were because we started with the Whisper encoder as the starting point, and now we have in-house encoders. Like the bigger time model, for instance, which we released in January, we also released a technical report for that real-time model as well, which is this dual stream architecture.

It's an in- an interesting architecture. You should check it out. And there we have a causal encoder, and I don't think there's any strong multilingual causal encoder out in the community, so we thought it's a good contribution, so that's one nice encoder. If the other people want to adapt, that's a good encoder, and we train it from scratch.

I think our full stack is now mature enough that we're able to train super strong encoders and some of these considerations like this padding and stuff is a function of the Whisper encoder, and n- now that we train encoders in-house, the design considerations are different. And for the question on text to speech, I think that also leans onto the original autoregressive decoder backbone. I think it says we're a- almost identical considerations.

I think the long context in, it's not even long con- So the model processes audio at twelve point five hertz, so one second maps to like twelve point five tokens. So I think one minute is like seven point eight tokens. You can get like up to ten minutes in eight K context window. Mm. And you can get half an hour in thirty K context window. So that's... And thirty-two K context is something that we are very comfortable training on.

We can extend it even much longer, one point eight K. You can naturally see how it can extend to even hour-long generations. Yeah, we need the like data recipe and the whole algorithm to work coherently enough through such long context, but the techniques are some way very similar to the text long context modeling, and the key difference is it's just doing flow matching autoregressively instead of a text open prediction.

Okay. I think that was most, most of the sort of voice questions that we had, but- I have a big question on- Mistral Small? Mistral Small. Let's go. So what is Small? How do we define Small? [laughs] What is this? What is this? I remember the days of Mistral 7B on my laptop. [laughs] It's not fitting on my laptop. That could run on the big laptop, but- It's just a different question of terminology.

Like here, but really by Small is enough active parameters. But it's true, we really glad we made another name. But yeah, we could have called it Medium, but then I think there's just stuff [laughs] with the large models of this, uh, I suppose. But yeah, it's a model that we really a mixture of experts. It's a model that combines different model.

Before, w- so we were doing the same because we had one model, general model for Mistral doing instruction following, where like a separate model that was Devtral. So really good at coding, specify specific to code. We had another model, for example, Magistral. So these were separate artifacts built by different team at Mistral, and now what we are doing is basically merging all of this- Mm... into one.

It was, even Pixtral was the first vision model we had was like a separate model. And the way we do things internally is that we are one team focused on one capability, build one model, uh, and then when it's mature enough, we decide to merge this into the main picture. But here it was the first time we basically merged all of this into one. There are some other things we didn't have time to merge in time.

For instance, like more capabilities or function coding I think would be, uh, it's going to be mu- much better in Mistral Small proper form. But, but yeah, for our latest model on the we're working on, also larger version of this. And yeah, key things is, is very sparse, 6B active, pretty efficient to serve, two fifty-six K context. Yeah.

I think what's interesting is just this general theory of developing the individual capabilities in different teams and then merging them. Where is this go- gonna end up? [laughs] Um- Like- Like we've seen the five things put together in this. Yeah. What are the next five team? I think actually OpenAI has gone away from the original 40 vision of the Omni model.

That's with what they were selling, all modalities in, all modalities out, but I feel like you might do it. I think there are some modalities where it's not completely obvious. For instance, for audio. For audio here, if you want to do transcription, I think it makes no sense to use a model this large. If you just want to tr- transcribe tech, it's, it would be very inefficient.

If you want to do audio, you probably just want to do the 1B or 3D model. Performance would be essentially the same. It's going to be incredibly cheaper. So here that's why we want to have a, a separate model that just does this. Yeah, I think the question is just, yeah, if you are talking to your model By speech, and you're asking like a very, very complex questions and how you do this on the other prompt questions to cascade things.

Do you want to put or do in a model that has like a one key in around it? It's like a- Mm. Not a competitive discussion, I think unaware if you keep going into that direction, but that's the possible approach. But yeah, but I think for us the next capabilities we want to try to integrate into these models right now are going to be, yes, like more coding, more, more reasoning.

But I think more capabilities that people don't talk too much about, but it's very important I think for our customers in our... on different industries. For instance, things are around like legal, finance, computer-aided design, a lot of these things that is these models out of the box are never to put at that because people really don't prioritize this.

There is no like too many benchmark on that. Mm. But it's not hard to make this model good, and it's just have to do the work, like sourcing some data, processing it, keep increasing it into the next version. So, so yeah. But we have other things we merge into this. I think for voice, yeah, the key thing, I think over maybe like the last year or so with VO and Grok Imagine and all these things is joining voice with video, right?

Which people don't understand spatial audio because like most TTS is just, "Oh, I'm speaking to a microphone in perfect studio quality." But when you have video, like the voice moves around. [chuckles] That's true. The configuration was a little different in the sense that there it's like a, a standalone artifact where you get the whole thing and you consume it.

But in the conversational setting, it's a, you need the extreme low latency- Yeah... streaming- Oh, yeah... would be one of the primary concentrations and- You can build a giant company just doing that. Yeah. [chuckles] So you don't need to do the voice there. But I was just, you know, on the theme of merging modalities, that is something I am like, wow, like I didn't...

Everyone up till, let's say, mid last year was just doing these like pipelines of, okay, we'll stitch a TTS model with a voice thing and a lip sync thing and what have you. Nope, just one giant model. [chuckles] Yeah. I have a two-part question. So one is, it's still open. It seems like open source is still very core to what you guys do, and I just have to plug your paper. The end 2024, this is the- Oh. Next to all the experts, like very fundamental research on how to do good MoEs paper comes out. Very good paper for anyone. That's just side tangent with- No, this thing caused- We bring that... 8x22 was like the nuclear bomb for open source.

I think it takes 7B more. 7B more? 7. Okay. Yeah. But this is a bigger 7B. Yeah. Yeah. I do remember this. I remember-- I don't think it was January, right? It was like NeurIPS. It was- It, it dropped during NeurIPS, and everyone at NeurIPS was like- It was December of '23. But I think, yeah, the model was out. The date is wrong. Yeah. I think there was a bit over. Yeah.

It's just a little update, probably. Yeah, no, but you have a point to make. No, you gotta check that. But then I just wanna hear more broadly on open source for you guys. And- Yeah... when you had asked earlier about what's next, what are the other side tabs working on, you, you put out LeanStral. This- This one is a surprise. This one. I, I was like, I don't...

This doesn't fit my mental model of Mistral. Yeah. First, for open source in general, I think it's really something which looks to the DNA of the company. I think we started it around this. We-- So we have an open-sourcing model since the beginning and even before this, so before this. So me and team were at Meta, we released Llama, and I think what was really nice to see that before this, for most researchers, like universities, it was impossible to work on LLMs. There was no LLM outside.

And if you look at many of the techniques that were developed after, for instance, Llama was open source, all these post-training approaches, like even DPOD, like performance optimization, all of these were done by people that had access to this model, and it would have been impossible to do without this model.

So it's really making science move faster. So we really want to contribute to this open source ecosystem. I think like the DeepSeek and also like very lot of impact. All these papers that are, I think, in the open source community are really helping the science community as a whole to move faster, so we want to contribute to this ecosystem.

That's why we are releasing very detailed technical reports of Mistral and our first reasoning model. And version of the reasons, things that worked, things that did not work as well. That I think helpful. On the... Yeah, so for the audio model, we're also going to share a lot of details, share a lot of them for the real-time model. On the... Yeah.

So we really want to continue this, basically belong to this community of people who share science. I think we really don't want to be living in a world where the smartest model, the best models are only behind closed doors, only accessible to a few companies that really have the power to decide who can use them or not.

I think it's a scary future we don't want to live in. We really want this model to be accessible to anyone. We want intelligence to be used and accessible by anyone that can use it. So yeah, so that's why we are pushing this mission and source model on the... Yeah, so not... So yeah, we've started, yes, so it's open source, not the first model, so not the best.

On the, yeah, LeanStral, I think is also one step into this direction. So it's, yeah, a bit different than what we are usually releasing. But we have a small team internally working on a formal proving, formal math. So I think a subject we care about in general, and we were working on reasoning. I think we started too early before LLMs.

Doing reasoning without LLM is very hard, especially when you work with formal systems, because the amount of data you have is negligible. It's a very small community of people writing like formal proofs. But the reason why we like it is because I think there is-- If you look at what people are doing with reasoning, these are-- the problems that you can use are usually going to be problems where you can verify the output.

So for instance, all this AIME problem, where the solution is a number between one and like a thousand, so you can verify, you compare this with a reference, or it's an expression, you can actually compare the output expression, generate your model with a reference. But there are many, most of the math problem and most of the reasoning problems, there is no like way to easily verify the solution. If the question is, show that F is continuous, you cannot compare any reference, right?

If it's like prove that this is true or prove this property, there is no way to-- You cannot like simply verify the correctness of your proof, so it's hard to apply the... There is no verifiable reward here. So what you could provide is, of course, like a judge, an judge that will look at your proof, but it's very hard and it's very-- you could also have some water hacking happening there, so it's difficult.

You could provide like a reference proof, but then there are also many ways to prove the same thing. So if the model is given a negative reward because it's a different proof, but maybe it was still a legit proof, just different, so, so it's not going to work well. What's nice with Lean and with formal proving is that you don't have to worry about this whatsoever. We just- They all function... As long as it compiles in Lean, it's functionally the same. Exactly. It's like a program.

If it compiles- Yeah... in Lean, it's correct. Yeah. It's very easy, and you can apply this on any kind of- It's just way too small, so no human will actually go and do it. [chuckles] Yeah. Exa- that's exactly. So only people can do it, it's like a very small community of people doing a PhD on that, so it's super small. On this side because it's actually very useful on not just math, but also in software verification.

So for instance, software verification today, it's a tiny market. Very few industries work on this, and we need that. It's usually going to be like companies like building airplanes, robotics, like things where they absolutely want to be sure life depend on this. But it's very rare that people formally verify the correctness of their software. But I think one reason for this is simply that it's just super hard to do. Are you thinking of TLA+?

It's the language that some people do for software verification. No, I am- Hard... most familiar with that with Coq that people use in, uh, in France, but... But yeah, it's the reason I think why people don't use it more and why this industry is not as big as it could be is because it's very hard. But now with coding agents that are there, it's going to be very different, and we're going to see much more of this.

So I think yes, industry there is going to be much larger in the future of those that we have these models. So yeah, here also anticipating just a little bit, we wanted to work on that because it's proving like a math theory and proving like a function essentially is the same tools. Yeah, yeah. One of my theories is that because the proofs take so long, it's actually just a proxy for long-horizon reasoning and coherence and planning maybe. A lot of people will say, "Okay, it's for people who like math.

It's for Lean. Okay, it's like a niche math language. Who cares?" But actually, and you use this as part of your data mixture for post-training and reasoning, actually it might spike everywhere else. Yeah. And I think that's un- underexplored, or no one's like really put out a definitive paper on how this generalizes. Yeah, absolutely. And I think even that's what we are seeing already.

For instance, if you do some reasoning on math, then the model can also do reason and code very well even- Yeah... if you just code that in the, in the earlier stage. So indeed there is some transfer, uh, some sort of an emergence that happens. And I think some-- It's also interesting. It's not just, I think, the topic in general, but it's...

There is a lot of connection with this on encoding agents because sometimes the model can see like a theorem that it has to prove. It's very complex, but then it can take the initiative to say, "I'm going to prove this three lemma. I'm going to suggest three lemmas, and I'm going to in parallel prove each lemma, so three of them in parallel with sub-agents, but I'm also going to prove them in theorem as three, the three lemmas are true."

So you can do this sub-agent approach. So pretty interesting. You can even if you fail to prove one of the lemma, you can actually maybe you'd succeed to prove the other lemma too, so you get some, possibly one here. So it's a bit less sparse than if you just get to zero one for the entire thing. So it's pretty interesting. I think we can actually unpack stuff here. Yeah.

It's also an interesting case just for specialized models in general, right? Like the cost thing you show is pretty interesting. Yeah. Similar score-wise, you're thirty, seventy, a hundred and fifty, three hundred bucks comparison. Yeah, small model. I think cost is a bit unfair, right? Because this one is at like inference cost.

This one is their on top with their margins on top of it, but we don't know anything else, so [chuckles] we gotta, we gotta figure it out, right? I did wanna actually push on that more, not on cost, but you mentioned about, okay, it's a great way to have verifiable long context reasoning. What are other frontiers that I'm sure you guys are working on internally?

There's a lot of push of people pushing back on pre-training, scaling RL, pushing compute towards having more than half of your training budget all on RL. Where are you guys seeing the frontier of research in that? You mean with o1? Just in foundation model training in the next... One thing that you guys do actually is you do fundamental research from the ground up, right?

So you probably have a really good look at where you can forecast this out. Yeah, but I, I think for us, we are still working a lot on the pre-training side. I think we are very far from a situational pre-training. I think ML4 pre-training will be like big step up compared to everything we have done before, so we are pretty excited about this.

And I think on the RL side, I think now we have more and more to think about this algorithm that will actually support these very long trajectories. I think when it was, for instance, JOPU, for instance, doesn't really work with any bit of policy, which was okay initially because you are solving math problem that can be solved in like a few thousand tokens.

So the model can like regenerate them pretty quickly. So when you do your update, the model is never too far off. Still, it's never too far off. But now when you are moving towards this kind of problems where something takes hours, like six hours to get a reward, then your model is completely off policy, so you have to actually build your accurate data.

A new infrastructure that supports this, but also new algorithms. So now everything we are doing internally, we are trying to build some infra that we actually anticipate this, uh, what we have in six months, one year, which is these extremely long scenarios on the path they said they can. I, I think when we started Mistral, part of me and maybe also Timothy, we wanted to this very nice environment where people are there, they can do research they like with a lot of resources. So it was nice.

I think things changed a lot when, I think, when ChatGPT came out. I think after that everything was very different. This time around it was the same again, but, but yeah, but it was nice, and I think we also want to work as part of this group chat we had before. Coming to the end, we're just-- Obviously, I think you guys are doing incredible work. You've laid out a very impressive vision for open source and for voice. What are you hiring for? What's the...

What are you looking for that you're trying to join the company? Yeah. So we are hiring a lot of people in our science team. We are hiring in all our offices, so we have, uh, our HQ is in France, in Paris. We have a small team in London, like a team in Palo Alto as well. Recently, we opened some offices in, in Warsaw, in Poland, so one in Zurich.

We also have like some presence in New York as well, and soon one in San Francisco, so we are a bit everywhere. Also like hiring people remotely, so we are growing the team, trying to hire like very strong people. I think we want to stay. So the team is not... Still a fairly small team, and I think we want to keep it that way because we, we find it quite efficient. We like a small team, very agile, so yeah.

Okay, let's focus on science and the for deploy. We actually are strong believers in science. We started the, our new science pod that focuses specifically on AI for science. What areas do you think are the most promising? What we are pretty excited about right now, and something we have already started doing, and we'll be probably be able to share more about this in a couple of months, is that we are exploring AI for science.

Um, there are a lot of areas where we think that you could get some extremely promising results if you are to apply AI in these domains. There are a lot of low-hanging fruits. You just have to find these domains where actually AI has not been yet applied. Uh, and it's usually hard to do because the people working in those domains don't necessarily know the capability of these models.

They don't know how well AI will- You just have to pair them with- Yeah, exactly. Researchers... matching, which is actually hard to do. But this matching, we are doing it to naturally with our customers. So we have some company we work very closely with. So for instance, uh, ASM and Treason are one of our partners, so we are doing some research with them.

And there are, there are like tons of extremely interesting problems, problems in physics, in science, material science, that they are essentially the only ones to work on because they are doing something whereas no one else is doing. On the... Yeah, so there are many domains where AI can actually revolutionize things. It's just you have to think about it and be familiar with what it can do well now and how to apply it.

So yeah, it's something we are more modeling with our, uh, partners, with our customers. So AI for science is quite a, one big thing. Yeah. Okay. And then for deployed, what it makes a good for deployed engineer? What do they need? Where do people fail? I think it's usually you need people that are very familiar with the tech, and not necessarily with lot of research expertise, but that are actually, uh, pretty good at using this model, that can actually, like, that know how to do fine-tuning, that know how to, like, start some RL pipeline.

And it's, it's not easy. It's something that most customer-- majority of companies will not be able to do this on their own. So here, I think we need people that are, that like to solve problems, that are excited about like solving some complex, very concrete problem. It's applied science, basically.

And, yeah, so I think it's not too different, I think, from the skills you need when you do research, because essentially you are trying to find solutions to problems that customers have not yet solved. Sometimes it's easy, sometimes you really have to do the work. You have to, like, create synthetic data, find some edge case. So it can be... Yeah, depends on the problem.

But, but yeah, you have to, I think you need also a bit of patience and be creative. I think very similar skill as a researcher. The diversity of the work they do, it always surprises me. It's, it's, it goes all the way from the kind of stuff they encounter in industries. It's just very interesting, I think. Any fun, like, success anecdotes? Yeah.

It can be like really training this small model on edge that will just do one specific thing. It can be like training some very large model with some specific languages as well, making models really good at some to-dos, like for instance, computer-aided design, these kind of things. Is that in pairing with vision as well? Yeah. And defect detection for chips or, like in, in factories identifying things. Like it... The diversity could be anything where you can deploy these foundation models. Yeah.

So the work to make it work in that specific setting, basically whatever it takes to make it, like, add value in that specific workflow. Yeah. And it goes across the stack, right? Like, even just pulling up the website, like you have- It's true. It's so broad... the compute is so broad. We didn't even touch on Mistral Vibe. We have a live coding CLI tool.

One thing you guys were actually like, I think the first tool was Mistral Agents. You had a agent builder, you can serve it via API and all that. I, I'm guessing Forward Deploy people- Yeah... helped build that out and stuff. It's also why we are... So we are doing many things, but I think that's also part of the value proposition that sometime new customers are always very extremely careful about their data, and they don't want to...

They don't like trusting so many partners. Trusting one partner for code, giving your data to another third party for like audios, and another one. So they don't like this. So here, what they really like with our approach is that we can help them on anything. Mm. Uh, so they don't have to send all data to so many clouds. So yeah.

I think that there can be many orders of magnitude more FDEs than research scientists, and they don't need your full experience, but they're still super valuable to, to customers. I mean, in practice, these two teams are still quite intertwined very often. Yeah. So first of all, they are using the same tools, the same data pipeline and everything. On the...

It's, it's very helpful for the science team to get the feedback and the solution team, because they can say, "Look, these customers are trying to do this. This is not working. Can we be sure that in the next version-" Yeah. This is basically a real world eval. [laughs] Yeah. Yeah, exactly. [laughs] It's a real world eval, and it's not something...

For instance, if you are just working in the lab, it's just ships model, but you don't do this work of deploying the model for customers. You have no idea whether your model is good at this edge case. For instance, you know, even in EuroBC4, this, right? Yeah. There is a very gap, big gap between the public benchmarks that are very like academic on the, the- The real cases are just very diverse, and- Yeah...

in the specific context of a customer, you can fine-tune and make it like first evaluate, but create a solid eval benchmark and then measure in the context of their... The kind of audios, like for instance, one use case is literally just there's the word for kids, and they have to just say it out. It's a very specific thing. You're just saying one word, and then you have to...

You- you'll, you'll grade the kid whether they did it right or not. It's like RL for kids. [laughs] But so there's very diverse use cases, and the idea is that the, the applied scientist engineers will go and make it better. And then from the learnings, we incorporate it into the base model itself. So it's, it's just better out of the box. Yeah. It's a good full circle system. Like, the foundation model evals are all just proxies of what you really need. You're never gonna have one that says...

It, it doesn't make sense for there to be a one-word transcription like that. It's not something you wanna fit on. Perfect. Everyone should go check out everything that Mistral has to offer and try the TTS model, which we'll link in the show notes. But thank you so much for coming. Th- thank you. Such a pleasure. Thank you, guys.

[outro music]

Mistral: Voxtral TTS, Forge, Leanstral, & what's next for Mistral 4 — w/ Pavan Kumar Reddy & Guillaume Lample

Kapitelklicken springt direkt hin

Shownotes

Transkriptvom Podcast bereitgestellt · Zeitmarken springen hin