Donato’s local AI journey: Strix Halo, Toolboxes and the community

Usman Pirzada Hey everyone, welcome to Strix Halo Stars, a podcast exploring the people, projects, and ideas behind the local AI Strix Halo community. This is actually Donato Capitella's podcast, but for the first episode, we're switching things around. I am Usman Pirzada, the Client AI lead at the consumer marketing team at AMD, and I'll be interviewing Donato about his Strix Halo journey, the toolboxes project, and what's happening in the community.

Usman Pirzada Donato, what were you trying to do and what led you to choose Strix Halo?

Donato Capitella Hey Usman so nice to see you again. We actually met in person just a few months ago in San Francisco. It was for AMD's Advancing AI conference that was such a blast and come to think of it, that conference is the reason for this podcast. That's where it came from because AMD had invited so many people from the Strix Halo community, so many of the contributors that I actually got to meet in person and I thought it would be so nice to launch a podcast which is a little bit less technical but actually shines a light on the individuals that are doing cool interesting projects and talks about their story and how they got to work on Strix Halo and what they're doing with the platform.

Donato Capitella And then you said, well, if you're going to do that, I'm going to interview you for the first episode. So thank you very much for doing this. And to your question about my own Strix Halo journey, oh my god, that feels like it started ages ago. But actually, it's only maybe less than two years when I started.

Donato Capitella I was already into AI, I had a small YouTube channel that nobody really cared about. I was trying to build an LLM quite naively because I didn't have any resource to do so. But then not many people probably know this, but I work for a cybersecurity company. That's my main job. People think I do YouTube full time. I don't. so what was happening is that I was doing some work for some of our clients and I wanted some local AI.

Donato Capitella It was becoming clear to me that some of the stuff I wanted to do, I needed to run it locally. And it so happened that Nick Jones, who is the director of research at the company I work for. And by director of research, I mean the person that owns a budget that I can spend. He came to me. This must be April twenty twenty five and said, Donato, I've got some money. Do you want some local AI? Do you want to buy some gear?

Donato Capitella And I'm like, absolutely. Now, I think we can be quite open about it. At the time, what NVIDIA had announced DGX Spark, right? And so I was trying to go for that, but it would not materialize for a very long time. Nobody knew and it was I imagine it was quite difficult for them to put it together because it's not the same as doing discrete GPU's. So it happened that I came across Framework and they were doing the Framework Desktop and all my spider senses were immediately triggered because I'm like, this is integrated GPU, unified memory, one hundred and twenty eight gigabytes of that memory.

Donato Capitella And I knew that the amount of memory was going to be important. I knew that for running models that would be meaningful, I needed that. So I get the budget approved and I order the Framework Desktop. I am in Batch eleven. Now, this is maybe April, May, Batch eleven was supposed to drop end of September. I am a fairly impatient person. And so I cannot wait and I desperately start looking for somebody else that's gonna that put Strix Halo into any of the servers and then HP I come across them and it turns out I don't know if I've ever said this publicly, but it turns out that I go to every supplier in the UK, asking them, I know HP is making a Strix Halo server, have you got it?

Donato Capitella And one of them said, you know, we just had some coming in. I think we had the first ones and I placed one of the first orders known in the UK for one of these machines. So it arrives to me sometime in June or July. And I was so excited. I had one week to play with it before going on holiday, Strix Halo.

Donato Capitella So I had done a lot of research, teaching myself about llama.cpp, local inference engines, what models you could run. And there was a community called Strix Halo Home Lab on Discord, which I joined with a lot of very capable people. Back then, Leonard was one of the more capable people that was sharing a lot. I think he had a preproduction framework unit for a few months.

Donato Capitella And so I start, you know, spending that entire week getting this up and running. it was the same time that GLM air dropped

Usman Pirzada Yeah.

Donato Capitella four point five, something like that. That was a very good,

Usman Pirzada It

Donato Capitella even by

Usman Pirzada was a

Donato Capitella today's

Usman Pirzada very good model.

Donato Capitella standards,

Usman Pirzada Yes,

Donato Capitella you remember it.

Usman Pirzada yeah,

Donato Capitella I get it

Usman Pirzada very

Donato Capitella to run.

Usman Pirzada clearly.

Donato Capitella It's huge.

Usman Pirzada Yeah.

Donato Capitella Get it to run, publish a video where I document all of these. And you know, back then, you know, if you go back one year and a half, the software stack wasn't as mature as it is now. So you

Usman Pirzada Yeah.

Donato Capitella had to play with it a lot. And for me, that was great because I love tinkering with stuff. I mean, I'd rather have something that I have to study, modify to get to get to work. But then I published that video I published the containers that I was using, all the different Vulkan, ROCm, all the different backends, how to get these models to run.

Donato Capitella And then I literally go on a flight to Italy for holidays. when I get off the flight and I start looking at my phone, there are so many notifications from that video. Like my videos didn't get many comments, didn't get much interaction, but that Strix Halo video in, again, I think it was the beginning of August twenty twenty five, it started getting so many views, so much interest.

Donato Capitella So many people were downloading the toolbox, were commenting, were wanting to try these out. A lot of people were waiting for a Framework Desktop. Some had the ASUS the people that at that point had Strix Halo, most of them were the ASUS ROG, the laptop, actually, the laptop

Usman Pirzada Yes,

Donato Capitella tablet,

Usman Pirzada it was a tab that actually was not dealing the full laptop it was yeah

Donato Capitella which was incredible to

Usman Pirzada which

Donato Capitella me, and

Usman Pirzada was incredible

Donato Capitella they were downloading

Usman Pirzada yeah.

Donato Capitella that. And so that started my journey. other than the ego boosting effect of having one video after two years of doing YouTube that actually somebody cares about, but that's a signal. The signal was clear that a lot of people in the tech community had been waiting for something that could run large models that they could experiment with and that could also work as a general purpose computer.

Donato Capitella Like I don't want to have to buy like something like a specific server. It would cost too much, but ultimately Strix Halo is still an AMD CPU as well. I can run any operating

Usman Pirzada Yeah

Donato Capitella system I want on it. If I decide I don't want to do AI anymore, well I can game with it. I can run Windows, I can do whatever I want. So it wasn't locking people into one use case. I mean, actually the ASUS ROG, a lot of the marketing around it, if I remember at the beginning, was about how good the tablet was for gaming.

Donato Capitella you know, there was so much framework saw the video Nira, the CEO of Framework, wrote to me on LinkedIn saying, You said you wanted a Framework Desktop. We saw you made an order. We're sending it to you a unit now. And I was still in Italy and I know that like, you know, he sent it immediately. I thought it was a scam at the beginning. I'm like, Why is he writing to me? But then I had two and I started clustering them and I mean, long story short, I think I have made so much content on Strix Halo learned so much and it's a piece of hardware, okay, but there is a community that formed around it that partially, I think AMD can take credit for having nurtured that community.

Donato Capitella Certainly the work that I've done in the past six months has been supported a lot by AMD. So that's been an incredible journey. I still can't believe that I somehow know something about local AI.

Usman Pirzada no, I think your journey has been absolutely inspirational. And since you're actually one of the earliest people experimenting on this Strix Halo, I think since well before August twenty twenty five, so I think your journey has not only helped the community learn as well, but it has been a very important source of documentation and sort of a BKC, if you will, so like a best known configuration on how to do things.

Usman Pirzada And you can always make the hardware, but it's really the community that ties a product together and that really establishes a platform. And I think the community found that in you. And that's actually one of the things that I want to talk about. So one of your projects that I think has had the most impact, at least like from our perspective, is your toolbox project. And that toolbox project is something that made it really easy for people to have data points, BKCs, and yeah, I would I would just learn to hear more about it.

Usman Pirzada How did you think of it? What was like the driving force behind that? What can someone do with it today? How was it developed? How people contributed? And yeah, I would love to peek behind the curtain and hear more about it.

Donato Capitella So I would like to say that I planned it and I knew what I was doing, But obviously it wasn't like that. So a few things lined up perfectly, right? I've always been for a very long time a Linux user for better or worse, but like at that time specifically, the Linux distribution that I was using a lot and I'm still using a lot, Fedora, was starting to enforce, not enforce, but like have this philosophy of containerizing everything.

Donato Capitella This idea of like having Docker containers, but then another layer on top, the toolbox layer. So to say your base operating system is almost atomic. Like we give it to you. That's a tested build and everything else you want to run on it, it's a container, but this toolbox layer on top that you can add makes it seamless.

Donato Capitella So I use it as if it was on the host. And so because I had that, when I started experimenting with Strix Halo, I obviously couldn't install ROCm directly on the host. It would be very complex on that Fedora build because it's atomic. So I had to build containers. And then he made it easy because I could build multiple containers and experiment with the different backends, different compile options for llama.cpp, vLLM configuration settings.

Donato Capitella So I did it for myself, and then I published it and I saw because that I can see how many people started using that. And I think together with those toolboxes you were mentioning, which sometimes I forget, but I started benchmarking stuff. I

Usman Pirzada Yes.

Donato Capitella think that was important for me. Like if I have all of these backends, how fast is the model? If I compile llama.cpp with this version of ROCm and this configuration on this model, what's the token generation and prompt processing? And sometimes it was dramatic the difference that you would have. And so people started using those benchmarks to see which were the backends that would perform better on Strix Halo because obviously you want to try and squeeze as much performance as possible. I think there is no case in which you're going to say, I am going to settle for less performance because the models get larger and larger.

Donato Capitella And so I started publishing all of these. And because there was a community, the people were using it, it kind of pushed me into keeping them up to date. And AMD, there was a time, I think where something was shifting in AMD. I don't know this for a fact. You probably know more, but like between my first toolbox and like if you go one year in ahead of that, the ROCm stack became like so much better, not Strix Halo.

Usman Pirzada yeah and I can sort of show you the other side of the thing. I think that was January of twenty twenty six when we essentially released our first full Windows release and we went all out. There was a preview release in October of twenty twenty five, but January is really when things kicked into absolute hyperdrive.

Usman Pirzada But yeah, continue please.

Donato Capitella No, no, but that's but there was so much that AMD was starting to do on ROCm, and it made it easier for me to keep everything

Usman Pirzada Yes.

Donato Capitella up to dated, but it was also exciting because every time there would be a new version of ROCm, you would go in and recompile llama.cpp against that. And I even automated it. I had pipelines that would automatically pull in the latest version of llama.cpp, the latest version of ROCm and compile it and test it on what were the main models.

Donato Capitella And so I kept maintaining those and those benchmarks now they have changed a lot. I have a different methodology, but they still exist on local and benchmarks. But I do like prompt processing and decode curves at different depths in the context. But anyway, I think the mix of these environments and everything happening, so all the updates, which meant I had a reason to keep the environment, the toolboxes up to dated with the latest versions of ROCm, llama.cpp, and new models that were dropping.

Donato Capitella So I had a reason to redo all the benchmarks and people started using that. And I think that's what made that project I think really, really useful to people. I know that those benchmarks were used a lot and again, now they're better. They had some issues as well, but if to answer your question in TLDR, it was the mix between having stuff that made it easy to try different options, not just to try because you could do LM Studio, which works fantastically.

Donato Capitella I do recommend it if you like want to get started, why not? And it also became much better as Lot Studio came out. There is so much there. But if you wanted to tinker and try all the different back ends, different kernels, different patches and test how fast they were, this is the kind of work I was doing.

Donato Capitella And I imagine some of the stuff that I and the community because obviously it was a public repository, so people started making PRs and everything got, but when people started giving me feedback, I mean, in the community, some people are exceptional. Like the level of expertise is incredible, and that kept making the project more and more supported.

Donato Capitella And I think a lot of those learnings that came from the community, which I incorporated in the toolboxes, but other people also contributed and incorporated into all the other engines. And again, I think everything LM Studio, Ollama, everything is better out of all of this collective work that people have been doing.

Usman Pirzada yeah, that's I think the development journey for the MD ecosystem, as you say, has been absolutely incredible. And yeah, we have been so lucky to have amazing partners like you and on the ISV side, LM Studio has supported AMD since our day one of Strix Halo basically, which was back in, I think March, twenty twenty five on shelf.

Usman Pirzada So yeah, partnerships are what makes the dream work and we have been very lucky in that regard. And honestly, the entire local AI ecosystem has been ramping up so fast. Like things have been so exciting lately. Like you get these new model releases, you get these new frameworks and the developer community is just absolutely amping up. Like, for example, GLM five point three flash, wow, what an amazing model. And Qwen three point eight flash next is probably right the sweet spot of parameters and intelligence right now.

Donato Capitella Just a quick pause to thank AMD for sponsoring the first season of Strix Halo Stars and helping me make this podcast possible. Now back to the conversation.

Usman Pirzada And it's not just the models that have been blowing up. We are seeing custom inference engines, custom formats. So for example, ROCmFPX Helogen has been incredibly popular on Strix Halo. Now there is Gufo, I'm not sure if that's pronounced Guford or Gufo, but yeah, Gufo is amazing as well and I think it even has a Windows port now.

Usman Pirzada So like the amount of developer ecosystem that's sort of coming around and growing around Strix Halo is incredible. And I'd love to hear your thoughts. What do you think is driving that development?

Donato Capitella So yes, I think there has been so much happening in the community. If you just go back to again, one year ago, you had llama.cpp, maybe vLLM, and there was a certain way of doing this. You had one inference engine and you tried to make that inference engine with kernels that were general enough that will allow you to run any model on any architecture. That's basically the llama.cpp design process.

Donato Capitella Now what's changed since then, which I would have never really expected to be possible to that scale, is that thanks to AI, what's become possible is that people that know what they're doing can now make either a fork or a complete new engine, completely from scratch to target a specific GPU architecture.

Donato Capitella In our case, Strix Halo gfx1151. It's got its own specific instruction set, amount of memory, memory bandwidth, size of the cache, so all of that to a specific model architecture for the inference on that specific model architecture, sometimes to a specific quantization. So with using specific approximations and operations, and that's what kind of it's a breath of new life into Strix Halo because now that we have obviously a few models which are winning in terms of a few model architectures which are really good.

Donato Capitella So you don't need to run a thousand models. There is probably five open weight model architectures, maybe even four that you want to support at the moment. So

Usman Pirzada Hey

Donato Capitella if you can make engines specific for that and then optimize that architecture, you're going to get incredible performance. Like everybody, I think was very surprised when Helogen came out, for example, and you looked at the prompt processing that it could do and the token generation.

Usman Pirzada now.

Donato Capitella I mean, that was three times as much you would get in llama.cpp. Not because llama.cpp is bad, llama.cpp is amazing. Think about it. It's the only engine that can run any model anywhere. If you have a CPU, if you have Strix Halo or if you have a discrete, I mean, it's incredible. But obviously, the tradeoff is that your kernels, your architecture has to be generic. It cannot specifically target things. But once you can make, with the help of AI, because this is a lot of work, and that's where I say AI has really helped this generation in the last six months of all of these very specific engines.

Donato Capitella You were mentioning Gufo. I spoken to the two guys that make Gufo. And again, it's this idea of choosing a subset of models and targeting the inference to that architecture and playing with very specific parameters, the shapes of what the GPU kernels take in to minimize the use of the memory or to maximize the use of the memory bandwidth versus the compute units to find what's the best possible balance, what's the best operation that the architecture can do?

Donato Capitella Strix Halo as WMMA units. How do we best use them? They can take tiles of sixteen by sixteen in certain formats. So how do I look at the architecture of these models and what's the best quantization that I can then target without losing that I can target to the instructions without losing the quality because obviously I still want the quality.

Donato Capitella And so all of these engines came out and specific quantizations, like I know Carlo, for example, works on the Rock MP. So FP four, FP two, FP six, and that's again another way of looking at what is this architecture specifically offering me in terms of instructions, in terms of data types that it can work with.

Donato Capitella And how do I create a quantization that strikes a balance between quality or adherence to the original model and then speed of execution on the architecture? And that's made possible, I think, so much by the use of AI, which allows you to do a lot of these. Myself, I helped develop DwarfStar, which is yet another of the engines. It's been one of the first engines implementing DeepSeek before when it came out. I worked a lot on the ROCm version of that, but I used AI as well.

Donato Capitella So I started creating a method where I was profiling or asking AI to profile each of the kernels in the architecture. And then the way that now the method works by getting AI to generate a thousand different variations of a certain kernel in certain shapes, profile that the use of registers, the memory bandwidth that it's been used, the amount of operations, the precision that you can get trying different operations, and then picking the kernels that work the best and promoting them into the worst.

Donato Capitella As a matter of fact, right now, I am trying to improve the performance for another model, DeepSeek V4.1 Flash on Gorgon Halo. And again, this process would have been very expensive before. It's not that we couldn't do it before. We've always been able to do this, right? But it would have taken an amount of time that very few of us had.

Donato Capitella I mean, AMD has that kind of time and expertise, but outside of companies like AMD, there is not many people that would have had that expertise. And now this is easier for us to do. Now, on one side, putting myself in the shoes of people approaching platforms like Strix Halo, it can be confusing. What do I choose? It was easier before.

Donato Capitella I say, Listen, do llama.cpp, maybe do vLLM, SGLang but like this is what's available. Now, I can give you a list of ten engines. And so I'm still doing benchmarking and obviously on the benchmarks, some of them are better with some models, some of them are better with some other modest architectures and quantizations.

Donato Capitella So on one side, it's confusing, but on the other side, the amount of performance we are squeezing out of the same hardware that we had one year ago is incredible. I mean, it's like buying a fiat panda and the same car the year later performs like a Ferrari because of everything that's changed around it.

Donato Capitella I mean, this is what it feels like if you were actually one year and a half ago using Strix Halo for local inference and you're using it now with any of these engines and obviously the new models that came out and all the improvements on the ROCm side as well, It's a completely different world. And so that's been one of the most exciting things that I've seen happen in the community. And to mention a few more things, if you go to the Strix Halo HomeLab Discord channel, there is now a community maintained fork of llama.cpp which is specialized to run as fast as possible on Strix Halo with a lot of people contributing to that. I think it originally came from Natan and a few other users that had like a Vulkan fork and then a ROCm

Donato Capitella fork with specific optimizations, and then the community wrapped all of that into the community fork of llama.cpp for Strix Halo. So you have individual engines like Halo, you've got forks that implement, again, ROCmFPX and specific quantizations, engines that implement specific kernels, you have Gufo, Strata, and I am probably forgetting a million other ones and there I can expect the next few weeks will have more coming out.

Donato Capitella This is fantastic for the entire community.

Usman Pirzada Yeah, yeah, absolutely. And you know, I think what you said about, you know, buying a Miata and having a Ferrari a year later really, really resonates because so sneak behind the curtain. I remember October November of twenty twenty four, we were like the entire team was preparing to launch Strix Halo at CES twenty twenty five.

Usman Pirzada And I remember the team going, okay, so how can we show this unified memory platform that we are launching for the first time in the industry and what can we really do with it? And I remember running Llama seventy billion at that time and the entire team sort of playing around with it and we were asking ourselves, okay, it's great.

Usman Pirzada You can run Llama seventy billion, but what can you do with it? And I remember Llama seventy billion pretty much the most you could do with it was basically summarization. It wasn't really good at coding. It could maybe like write a couple of lines of code or write a very simple HTML page. But I think all of us understood that, you know, and hoped for that once, you know, it's sort of like, you know, you build the kingdom and they will come. So the developer community is like the entire credit goes to the developer community for basically you have the same hardware now one or two years later, but now the amount of intelligence that can run on it is just absolutely out of this world.

Usman Pirzada And most of it is actually coming from these model vendors. And I think like if I look back at the last, you know, years, year and a half and how the models have evolved, Llama seventy billion was the pioneer, but it didn't really have the intellect. Then I remember, you know, the Qwen models dropping, but they weren't really ahead of the curve.

Usman Pirzada I remember Mistral for the great, for the longest time, if you wanted to run the smartest model, you were running a Mistral. Like, you know, there was then Mixtral came around and they actually invented the MOE, by the way. So, you know, it like people, people don't remember that. People credit that to DeepSeek, but MOE was actually a mixture invention with their MOE model.

Usman Pirzada And then, you know, the Qwen model started really getting really, really good. I think earlier this year, one of the biggest seismic model drops that still holds up today was probably the Qwen three point six thirty five Buzz the thirty five B MOE, that was the first time users could experience legitimately cloud quality experience locally.

Usman Pirzada Sure, it was cloud from six, seven months ago, but nonetheless you got that quality and then it has just been taking off exponentially. And you got the Qwen three point six three seven B now the Qwen three point eight family, the Qwen three point eight flash next and the DeepSeek is absolutely phenomenal.

Usman Pirzada GLM five point three flash is absolutely phenomenal. So how do you approach this? And if someone has a Strix Halo machine today, which models would you recommend that they try?

Donato Capitella I mean, I was thinking so many things when you were talking about the models because we lived through it and we almost don't realize that this is one year that we're talking about right? I remember the Qwen three point six moment. Qwen three point six thirty five B was such a big change. I think the Qwen team has been the king of fast, useful local AI.

Donato Capitella then Qwen three point six thirty five B comes out that is fast, that is fast on Strix Halo. And for a long time I thought, How is anybody going to be finding something better? And then obviously we had the DeepSeek for moment, I really like DeepSeek. I mean, in my mind, you have Qwen, which is the king of giving you the smallest models that achieve some of the most incredible stuff for their size. Like the Qwen models punch above their weights. DeepSeek, I see them as the research focused people.

Donato Capitella So pioneer, how much can you squeeze out of the same hardware. How big can you go? How can you save memory? How can I change the architecture? How can I compress my KV cache? How can I do all sorts of like tricks to go faster? And obviously we had GLM. In a way, GLM is a much less sparse model than DeepSeek. So

Usman Pirzada Yes.

Donato Capitella it's a bit slower, but although it doesn't have these incredible architectural changes, from the capability point of view, if you have GLM five point three, it might be a bit slow on Strix Halo, but you can actually give it an extremely complex task. So we've had all of these models coming out and they have really made a difference. What I recommend, obviously now, it's what you're saying essentially. I think right now, if you have Strix Halo, sixty four gigabytes of memory, I mean, if you have thirty two gigabyte version, Qwen three point eight twenty seven B is still going to be probably one of the best things you can run.

Donato Capitella If you have sixty four gigabytes, a hundred and twenty eight Gigabytes, then you can push to Qwen Flash next three point eight. It is the best compromise between quality and speed. That model

Usman Pirzada Yeah,

Donato Capitella is

Usman Pirzada it's

Donato Capitella fast.

Usman Pirzada a sweet spot. Yeah.

Donato Capitella You run it on something like Helogen or Gufo. I mean, we're talking about prompt processing in the order of one thousand six hundred tokens per second. I mean, shall I repeat a model of that quality that with these engines, you can process like that. And obviously, you know, it's because they're doing a lot the quality of the model comes from exploring all of these alternative ideas, you know, using ngrams. Now everybody has

Usman Pirzada Yes.

Donato Capitella understood, okay, for local AI, having this ngram table is great. But then we have, again, GLM is another model that I would definitely consider on Strix Halo. I would consider it as well if you want to run like a task that takes a little bit longer. You want to leave it unattended, but I know that Carlo has been doing some work on our ROCmFPX quant off that model, which actually runs much faster than some of the standard quants.

Donato Capitella But then again, I think DeepSeek is definitely an incredibly good model. And then DeepSeek V4.1 Flash, if you have two Strix Halo machines, obviously that's a half a trillion parameter model, it is bigger. Like we forget they call it flush, but this is

Usman Pirzada Yeah,

Donato Capitella half

Usman Pirzada yeah.

Donato Capitella a trillion parameter model, but if

Usman Pirzada Yeah.

Donato Capitella you have two Strix Halo machines, you can cluster them together with something like the DwarfStar, which is a lot of the work that I've been doing. This model, especially the update they released last week, what excites me the most about that architecture, first of all, if you are lucky like me and you have Gorgon Halo, it runs on Gorgon Halo.

Donato Capitella You need one Gorgon Halo, a hundred and ninety two gigabytes and that will run a half a trillion parameter model of the quality of DeepSeek V4.1 Flash. But you can do it with two Strix Halo if you want. What excites me about the model is again the architectural solutions. That model can run the prefill, so the prompt processing, you can run half of the weights for processing a large prompt, which means that right now in the optimizations that I'm doing, you can almost get to five hundred tokens per second of prompt processing, which is an an incredible amount for a half a trillion parameter model because you only have to activate half of the weights.

Donato Capitella That's one of the innovations they have in that architecture. And then with the last update, it's much more token efficient. So I tested the first version and now this model on Terminal-Bench Mini, which is an agentic encoding benchmark. It is better in terms of its ability to complete tasks. Obviously, it's smarter, the tasks that it could already complete, now it is much more token efficient, around fifty percent less generated tokens, which really matters because this is one of the directions I'm more excited about.

Donato Capitella These local models, if they can now be tuned not to produce too many tokens when they're thinking, especially the thinking. The thinking is really expensive because obviously you are sampling tokens one after the other and speculative decoding doesn't help much with thinking. Speculative decoding helps you a lot after the thinking when you're generating code or when you're generating something structured. But PC point one as of last week checkpoint does really well, it uses less tokens to think, the same quality of thinking, but it is not sampling a lot of weird stuff.

Donato Capitella It is not self doubting itself as much. And so ultimately, it is much faster. It feels much faster. It is the same speed of generation, but that's like a direction that I'm very excited about. And I want to see all the upcoming local models. Like I know that Qwen 4 is you probably know more than I do. I imagine you have information about when Qwen four is going to drop, but you're not going to say, but I think it is close. And so I'm very, very excited, not just about the models becoming smarter, but the models becoming more efficient because all of a sudden, twenty tokens per second in generation become perfectly fine for that level of intelligence if you don't have to produce a billion tokens before

Donato Capitella you can find the correct answer. So so much is happening and honestly, I think we are still like we were not even close to having saturated what's possible to do on these platforms.

Usman Pirzada Cool, yeah, thank you so much. This has been incredibly informative And it's just incredible to hear your entire journey. One thing that I really wanted to touch on before we wrap this up is what's next? Like it has been an incredible journey, but what's next? Like

Donato Capitella That's maybe a couple of things. What's next is a lot of practical work on Gorgon Halo to improve inference. Right now I'm working, as I said, on DeepSeek V4.1 Flash to try and harness this model to make it as fast as possible. And I'm working on something that I'm hoping to release very soon, which is my setup of how I use these models for coding inside containers and what kind of how do I make them as efficient as possible to actually complete a lot of tasks.

Donato Capitella So I'm very excited about that and I'm doing some testing and I will be releasing it soon, but there is also something else that I will share with people. I think this is not just going to be local AI versus cloud. I think you don't have to just make that decision. There is a world where you can get a cheap cloud subscription to OpenAI, Cloud, and Gravity, like twenty quid amount or something like that to get access to some of the frontier models and use them as advisors alongside a model like Qwen or DeepSeek.

Donato Capitella mean, a sneak peek because I'm still doing this experiment and I wish people understood how long it takes to do it properly. It's been running for three weeks, but I am taking tasks, complex encoding tasks that neither of these models can solve locally on their own, so that they fail five out of five times.

Donato Capitella The models fail these complex tasks, and I'm giving them Codex or Claude or Gemini as an advisor, and I'm telling the model this is the quota, check the quota, check the models that are available and use them just when you have a problem or for triaging something. So very token efficient. And so far, basically we go from five out of five failures, inability for the local model to complete a task to five out of five successes with very, very little use of that twenty dollar quota.

Donato Capitella And I think this is going to show people another way where you don't need a five hundred quid monthly subscription.

Usman Pirzada Yeah.

Donato Capitella You only need the super ultra duper frontier model to get you out of very complex stuff. But if you're using that model to list the files in a folder or to do a simple edit in a source file, that is a waste. We have local models that are perfectly capable of doing a lot of stuff. So I'm still working on that. Again, these things take months to fully benchmark and to fine tune, but I will be releasing that for everybody that's got streaks and it will be like, you know, you can get it up and running and it supercharges what you can do.

Donato Capitella Together with the new models that are coming, I mean, this is some of the most exciting things that I'm working on.

Usman Pirzada That sounds incredible and looking forward to seeing your work. Thank you everyone. In future episodes, we'll be switching back the roles and Donato will be interviewing selected people from the Strix Halo community about their projects and ideals for local AI on Strix Halo. For more information, check out the Strix Halo Toolbox's website and Donato's YouTube channel. Thank you for listening.

Donato Capitella Thank you.

Donato’s local AI journey: Strix Halo, Toolboxes and the community

Kapitelklicken springt direkt hin

Shownotes

Transkriptvom Podcast bereitgestellt · Zeitmarken springen hin