AI Inference Is a Data Movement Problem: Microsoft's Saurabh Dighe on Maia 300

0:00 Hello, listeners, and welcome to another episode of Summitotes on Austin Lines with Chipstrat. And with me today is a special guest, Sarah DK, corporate vice president of Azure Hardware Systems, and he's an architect at Microsoft. So welcome, Sarum.

0:14 Oh, thank you. Thank you, Austin, for having me over. I've really enjoyed your YouTube videos, so I'm I'm glad to be doing this one as well.

0:28 Awesome. I appreciate that. So for newer listeners, um, you may not know, but Sarv and I talked back in January and we talked about the Maya 200 from Microsoft. And today he's back 10 months later and we're talking about Maya 300. Um so I guess with that, like remind us very high level again, maybe for new listeners, like why is Microsoft making the why did you make Maya 100, 200?

0:51 And then what are we talking about today with the 300?

0:54 Oh, yeah, absolutely. We'd love to do that. And it was a great conversation in January. So here we are, 10 months later. Uh before I just go deep into Maya itself, which I'm sure we'll do for the next hour, I think uh it's important for us to explain why Microsoft is building custom silicon and systems.

1:12 And we have this privilege of running a planet scale infrastructure. And we'll run a lot of workloads, both SaaS, PaaS, IS businesses, where we are able to understand what our customers' needs and our customer requirements are, and really to understand what infrastructure requirements do they need. So we are we are not here to just build a chip for chip's sake.

1:36 Like we're here to really differentiate and deliver value to our customers. And that philosophy for us is really not a chip story. It's a how does the whole system work together to deliver that value to our customers? And so uh we have been on um multiple years of silicon and systems journey, right? Inside Microsoft, uh, you may have heard about our compute silicon, which is cobalt.

2:01 Uh, we're doing Maya, of course, for our AI. We do Azure Boost for our net front-end networking, we do Azure DPU for storage processing and data processing, and then we have a lot of custom security silicon chips. So we have been embracing this philosophy of really building our infrastructure that delivers value to our customers and gives us that freedom to innovate so that that infrastructure can then become something that the customers can rely on at scale.

2:30 So now the Maya journey really started for us in Maya 100, as you mentioned before, but this was before the world of LLMs, right? We were starting to look at hey, what does it take to really accelerate uh speech and sometimes convolutional neural networks? Because that was kind of the growing uh trend that time.

2:51 And then we quickly entered the regime of LLMs when we decided to build Maya 200 for how do I build the best AI infrastructure for LLM inferencing? And and this is a bet we took because a lot of that time period, people are talking about training, training large models. And and we looked ahead and we said, even though training is gonna be super important, really the distribution of that intelligence is gonna be equally important, if not more.

3:20 And so this is where we took an inference-first approach for our Maya system. Again, I want to say it's uh it's not just the chip story, but it's really about how we kind of build the best AI infrastructure system and take the software, the orchestration that goes around it, the model family, the model families that need to be ported on, the custom silicon itself, the memory subsystem, the networking and data center, and really wrap the whole thing into how do we maximize the token throughput or the intelligence we can deliver for every watt or every dollar that we deploy.

3:58 So that's really the our philosophy behind the Maya architecture. And so we we disclosed Maya 200 uh earlier in the year. Uh today we are unveiling the Maya 300 architecture. And in the Maya 300 architecture, we we will talk about kind of the fundamentals of the foundations that we built on top, but then how we develop a generational leap in delivering the performance per dollar and the performance per watt uh with this new system and architecture.

4:28 Yeah, this is this is all great framing. I love that you started with planet scale because I think you know it's easy to forget just how massive Microsoft Azure's reaches. And therefore, especially when it comes to sort of the defining workload of our time, LLM inference, how big the opportunity is to co-optimize, co-design the workload from the software to the serving layer, you know, all the way down to the infrastructure, to the custom silicon.

4:57 And to your point, not only the accelerator, but the memory decisions, the interconnect decisions, as I know we'll get into scale up here after a bit. Um and so yeah, I can definitely see how uh this is a huge opportunity to co-design. Um and then, you know, we'll talk about workloads too here in a little bit, but but I think just to remind listeners, um, you know, even just uh first-party workloads for Microsoft of, you know, GitHub co-pilot, Microsoft co-pilot, Microsoft Foundry, you know, obviously you you guys are already deploying inference at scale.

5:30 And that's not even let alone, you know, maybe third-party workloads. Um, but maybe talk really quick at a high level about the workloads, and then we'll get into like the architectural differences with the Maya 300.

5:41 Absolutely, absolutely. And yeah, I think uh, as you know, since we talked in January and now, uh the shape of the workload and the computational profile has definitely changed, right? We are moving from predominantly a chat-based experience to now agents with long-running tasks. And that has definitely kind of rethinked how we build infrastructure, how we deliver infrastructure for this new agentic um uh computational profile.

6:09 And we are starting to see uh the number of tokens that every you know agent generates is a lot more than what humans can generate, right? We're starting to see more persistent KV because the context is getting longer. We are able to see that, hey, you know, the agent is planning, then doing tool call, then ingesting new documents, then running an inference and iterating that loop.

6:33 And so KV is getting more persistent. We're also starting to see how important it is for memory bandwidth and memory capacity to be kind of a first-order design principle. We're also starting to see how important it is to pack more and more efficient compute in a given power and capital envelope or energy constraint and a capital constraint.

6:57 So all of these changing trends and changing factors we are able to see firsthand with the workloads deployed at scale and how customers are using it and being able to get that telemetry out of the systems. At the same time, design the entirety of the system to then maximize the intelligence you can deliver in that constraint.

7:18 So, yes, workloads are definitely, definitely changing. But I think with Maya 300, we feel very confident that some of the decisions we made are now we are able to extract the right cost point, the light latency point, the right performance, and think of it as the right class of service for that particular workload.

7:40 And like we talked about, you know, uh we tend to talk workloads, but they're not monolithic, right? It's almost like we we talked last time, there is there is this frontier efficiency curve. And we do believe that one size does not fit all. And so I think uh when we talk about Maya and some of the architectural decisions we made, we we make sure that the system was balanced, but it was also leaning towards where we believe is longer context, you know, memory bound, wrong learning workloads.

8:10 That's where I think we can extract the maximum value out of the system. And there are other systems that we will deploy, we will deploy a heterogeneous fleet to make sure that we cover every point on that frontier efficiency curve.

8:23 Yeah, yeah, yeah. No, that's good. And feel free as we talk through these architectural decisions, like you mentioned, um, you know, the workload now demands with the agentic like a long-running, kind of persistent KV cache, lots of memory, bandwidth, memory capacity, but continue to sprinkle in um your thoughts on how the workloads are shaping your design decisions.

8:43 Because, you know, when we see that Pareto frontier with um throughput on the y-axis and interactivity on the x-axis, I'm always interested in what the actual deployment and the companies are choosing for obviously they're not necessarily like filling out that whole curve. So I'm always curious like, what is the class of workload, like you said, you know, that would land at a certain point?

9:05 Is it you know higher throughput or lower latency? Um, so you know, again, as we'll go through this here, but but definitely feel free to sprinkle that in because I I don't think we get a lot of you know detail about exactly which points are people sort of optimizing for.

9:20 No, that's that's that's uh absolutely will do. I think one of the things that uh you know we believe in as we look at kind of the first principles of how you're designing an AI inference system, um AI inferencing to me is not a compute problem only. It's actually a data movement problem. And so we decided that instead of um trying to attack the compute problem, let's see how we can create a new class of architecture that really focuses on the data movement energy.

9:51 Like the industry has done an excellent job over the last few years with you know new microarchitecture, new process technology, and and and new nodes of and even new data types to actually be able to reduce the energy per flop. But what has not kept up is how do you reduce data movement? Because that is a significant, sometimes two orders of magnitude in terms of the energy that you could consume to move data and keep the compute engine fed.

10:23 And so one of the uh fundamental uh principles of our architecture is what we call the software-defined data flow architecture, which we introduced in Maya 200 and we continue to enhance. And and the underlying uh thesis is that with AI inferencing, right, we actually can plan for the data movement ahead of time.

10:45 And because we can plan for the data movement ahead of time, because we know what these data movement patterns look like, we can have distributed memories, we can actually pin their data at the right location at the right time, we can hide communication and computation uh and actually continue to improve the efficiency of data movement in this architecture.

11:08 And now with uh you know AI-assisted kernel uh development, it has become even more efficient. And we don't need armies of software people to be able to go and program this system, right? We can have agents optimize kernels. And so, to your point, uh the workload, if it's LLM inferencing at the heart of it, it is actually the data flow architecture we believe gives the best energy efficiency for inferencing for tokens.

11:39 So that's where we started, right? At the same time, we said, okay, that's great, but we do need to be able to now optimize the compute engine and the data engines together, right, in one system. And we then took an approach where we had distributed memories on the I, so we do use a lot of SRAM and we use it as a scratch pad, and so we expose uh the programmability and the explicit uh you know data movement instructions back into the compiler of the software layer.

12:14 We're also able to create uh much better computation and communication kind of patterns on DAI. So we have like the SRAM with our tensor unit, our vector unit, but then also we have a global knock that distributes the data within the chips inside the package. And so that really helps us again reduce the data movement.

12:37 Um we have gone towards a more uh NUMA-based architecture, which means that you know, once you've placed the data close to your compute, you don't have to drag it from another memory controller or five sitting from the other side of the chip. So a lot of thought has gone into how do you continue to keep the data movement, the data storage, the data computation at the lowest energy possible.

12:59 And then as we move outside the SOC, then we look at it and say, oh, okay, how do we connect these chips together? And a lot of times people think of this as, hey, I'm just gonna put a network uh and and connect these chips together. But we thought of the network as an extension of the data flow architecture.

13:15 What that means is that specialized data movement engines are DMAs and the communication protocol that goes on top with chips in other in you know, other nodes or other servers and other um racks can talk through the same communication paradigm as what they're doing on the chip as well. So it's just an extension of that data flow architecture.

13:41 And then finally, uh we also made sure that we drive a lot of rack scale efficiency. Because at the end, what matters is what is the deployment. People don't care about just the chip, people care about hey, how does the entire deployment deliver more tokens at a lower operating cost? And that's where our rack scale architecture really, really helps us build that into a more efficient deployment scenario in our data centers.

14:08 So, yeah, lots of layers of optimization, right? From starting from the silicon itself all the way to how do we deploy and allocate power in the data centers.

14:19 This episode of Semi-Doped is presented by Crusoe. They build the power, the data centers, and the cloud behind AI. Did you know Crusoe can help you tune an open source model on your own data for just a few bucks? Frontier models are very good, but they don't know your company's terminology, your part numbers, or how you like your reports formatted.

14:41 So you keep stuffing instructions and context into the prompt of an expensive frontier model to get the job done. But for simple repeatable work like reports, an open source model trained on your data can actually match a much bigger model. Think of a new grad with three months of on-the-job training.

14:58 You don't always need the PhD. And the open model is cheaper and faster to run. Fine tuning used to mean renting GPUs, working on orchestration and Kubernetes, writing training code, and babysitting the entire setup process. But Crusoe's serverless fine tuning removes all of that. Upload your datasets, pick an open model like GLM, Llama, or JeepSeek, and they handle the training.

15:23 You pay per million token used, not per GPU hour. And on the smaller models like Quen38B, $2 can take you a long way with millions of training tokens available. Once the training is done, you can then deploy them on Cruso Cloud in just one click or download the weights and run them anywhere. Your data only fine-tunes your model.

15:43 And it doesn't have to be work data either. You can try it on how you write emails. Pick a task, compare the tuned model to your current prompt, and see how it does. Crusoe's giving all new accounts $5 in credit to get started today. That's enough to start tinkering with fine-tuning right now. Links in the description.

16:02 Okay, so the framing is all with the Maya 300 and with Agentic AI, it's all about minimizing data movement. And so you're you're co-designing all the way down from the amount of SRAM on the chip to the local HBM to the software for developers to manage this, all the way to the networking and how you're communicating across chips, because of course on a scale-up fabric, they're sharing memory, so to speak, with DMA.

16:27 Um, all the way to rack scale, how many accelerators can we fit into this rack, and so on and so forth. So let's see, maybe for listeners who haven't like uh read the recent announcements, can you just zoom out and give us the high level like what does the Maya 300 look like? Like how many chips are in a rack?

16:45 Um, and and maybe so on and so forth. But then, and then we'll dive into uh the memory specifically. Like I want to dig into you know, the SRAM versus HBM trade-offs and decisions you made, and also maybe comparing against Maya 200, like HBM 3E to HBM 4. But but yeah, maybe just zoom us back out and kind of remind us at a high level, like some of the um, you know, specs for the Maya 300.

17:08 No, absolutely, absolutely. So what we've announced today is is really um our conviction of the Maya inference system architecture, right? We have Maya 200 uh in production, you know, outperforming some of the GPUs that are in our fleet today. And with that evidence and the production proven, uh, we are saying that we are going to be scaling Maya to multiple gigawatt deployments starting in 2027.

17:34 So that's that's basically a pretty significant strategic uh um positioning for us as Microsoft. And it'll be running, you know, it will be a foundation element of all Microsoft product and services in the future. Now, Maya 300 specifically, what we are saying is that over Maya 200, we are we are expecting you know this rack scale architecture to deliver 7x more tokens throughput per rack than what we could do at Maya 200.

18:02 And it will be delivered at 2.5x more performance per watt. So uh the interesting story now is how do we get there? All right. I think that's that's that's uh what we're excited to talk about. And as you look at it, uh we talked a little bit about like the software-defined data flow architecture. That's that's really a foundational element.

18:22 We also went to uh multiple chiplets, so we went to uh TSMC and two P process. Um, and with a chiplet architecture, we are really able to balance how much you know flops we want, how much HBM do you want in the system, and how much you know uh I/O or or scale up bandwidth do you want in the system. So we've kind of tuned the system a lot more to delivering, again, not stranding flops or not stranding resources in the system, but delivering that useful intelligence through every walk.

18:52 So it's a multi- multi-chiplet architecture really helped us there. In addition, we have introduced uh much better arithmetic, so low precision arithmetic, and we can talk about how with agentic use cases and with more persistent KV and growing KV, these you know, MXFP4-like formats really help us, you know, optimize our memory, like HBM memory, right?

19:16 So, you know, you like you said, our HBM memory is a constraint today in our uh environment. And you use it for storing weights as you are storing for a growing KV. And with lower precision data formats, we have optimized weight storage, model weight storage in for our lower precision. At the same time, we can optimize the KV with some of these lower precision.

19:38 On the compute side, we can have you know computation that's FP4 by FP8 so that we can continue to keep the accuracy and the quality of the model output at the same time, you know, pack more and more concurrency and higher KV in the same HBM. So a lot of thought in how do we uh use lower precision math.

20:00 Um a lot of thought in how do we actually design the memory subsystem. So, like we said, we have the on dye SRAM, but at the same time, we've also gone to HBM 4, and we have got eight stacks of HBM 4 on Maya 300 delivering a total of 18 terabytes per second of HBM bandwidth. That's that's that's almost 2.5x of what we did on Maya 200.

20:24 And if you look at the RAC scale level, that's 5x of what we could do on Maya 200. And so that really was a focus for us. Again, going back to how do we manage these longer contexts, more persistent KVs, you know, decode bound or memory-bound, decode-heavy workloads. Right. Um, our scale up network is the same scale-up network that we removed from right from a server all the way to superpod, which is now 12,288 accelerators in a superpod configuration or 96 accelerators in a pod, which we call as a rack.

21:02 So in a rack, we will be now 96 accelerators, which is double from what we did on Maya 200. And even though we double the number of accelerators, we can deliver this at 167 kilowatts per rack, which is quite extremely energy efficient if you compare it to anything else that you'll be uh getting from um you know other vendors in next year.

21:26 So 96 accelerators at 167 kilowatts, and then be able to scale up with one scale-up fabric, right, um, from rack to superpod to about more than 12,000 accelerators is really how we have spent our time man you know creating uh a really unique fabric, scale up fabric.

21:47 Yeah, fascinating. Let's get into the scale up fabric here in a second because it's it's very interesting to go so hard into the scale up fabric and make that a big bet there. So I want to unpack that. But first, the uh I want to hit The HBM. So, you know, and you've you've basically alluded to this, but I just want to hit it again.

22:05 You know, you went from six HBM three E stacks to eight HBM four stacks, and the bandwidth went up, I think you said maybe like two two and a half times. Um, and and capacity went up per accelerator. Um, so and I think you know, zooming out, the the amount of bandwidth and capacity went up more than the compute.

22:25 So maybe again, you know, it sounds like you were really making decisions here around um the decode side of things and the memory side of things for KB cache. Like how do you decide? Uh, you know, obviously, if you add more bandwidth and capacity, you can actually sort of unlock more of the flops if you're not keeping the flops, um, the compute sort of just waiting on the memory.

22:50 How do you thread that needle and decide, you know, how much more of your like die area you want to commit to flops versus um versus to HBM? And and also, I guess relatedly, I think remind me, do you guys have just one compute die with the eight HBM stacks around it? And and then maybe contrast with others who are doing like two compute dyes?

23:10 Yeah, I know I think this is a great question. And it really comes down back to you know, what problem are we trying to solve and why, right? And like I said, I think uh we continue to focus on inferencing as our primary. And so for inference-based workloads, we made want to make sure that we had a balanced system where we are able to expand our memory bandwidth and capacity, like we talked about, while keeping flops continuing to go increase.

23:37 And so we did move our and we did move to N2P because we believe that that's the most efficient process node for computation flops. Uh we went to a full reticle size compute, but at the same time, we felt that you know going towards two compute dies would have not been the right architectural choice for the workloads that uh and the shape of the workloads that we are seeing.

23:59 So instead, what we did was we still wanted the eight HBM stacks uh on HBM 4. And so this is where our chiplet architecture really came into play, where we were able to create these memory interface dye to extend the shoreline on the on either sides of the die. So we were able to extend the shoreline with chiplet architecture, put four HBM and four HBMs on either side of that while still keeping it as one compute, most very efficient, but one compute die.

24:25 Um and now going from HBM3E to HBM 4, you get those two multiplier effects. On one thing, you know, um picoJoule per bit is is super critical for HBM data transfers. And so with HBM3E to HBM 4, the industry has moved to a you know kind of a wider interface, but kept the kept the pin speeds kind of relatively the same.

24:46 And so what happens is you end up with a much better data transfer rate and energy profile for the data transfer rate. So that's one multiplier you get easier. And then the obviously the the second multiplier is when you in Maya 200, we went from six stacks, we had six stacks of HBM3E, so now we get to eight stacks of HBM4.

25:06 So if you put those two things together, along with our multi-chiplet architecture, you go from something that's delivering seven terabytes per second with um you know uh a certain amount of flops to now two and a half times that, so 18 terabytes per second, and with another around 35 to 37 percent of flop increase.

25:27 So I think the bandwidth per byte can significantly, or bandwidth per watt significantly improved over mile 200. And that's an interesting figure. So just with a 200 watt increase, again, we were we were extremely cognizant of the fact that when we put in the power into these SOCs and into our systems, it needs to translate to you know significantly better intelligence outcome.

25:56 And we did not want to strand any of that resources and on those resources. So those are those that's one of the thinking that went behind really kind of balancing the machine towards these workloads.

26:06 Got it, got it. Yes. In uh increasing your bandwidth per watt is very interesting figure, merit. And it makes a lot of sense and aligns with your earlier framing of you're thinking a lot about uh data movement. Um, okay, so I would for listeners, I want you to explain the need to expand the shoreline a little bit.

26:24 So you talked about obviously increasing more stacks. So instead of maybe having three on each side, now you have four. And then on top of it, you said the move to HBM4E doubled the number of pins. So instead of like 1,024 lanes, it's 2,048 or something. Is it just is it just a geometry problem, essentially, where now with four and with more pain uh pins, you just don't have enough room to route it all?

26:48 Or or why did why do you need the little extra memory die or whatever you called it to extend the shoreline?

26:55 Yeah, I think that uh uh just a quick correction. So this we're going from HBM 3E to HBM 4, not 4E. But uh everything else you said, absolutely correct. I think it's it's really two things, right? One is uh we are trying to optimize both for energy efficiency and cost. And so when you look at um energy efficiency, you get a much better picojoule per bit kind of a or overall uh transfer bandwidth per watt uh is is a better profile.

27:21 When we go to kind of the wider interface, you get a lot more bits coming out. At the same time, you also wanted to make sure that we are using the the right process technology for the right function, right? So if you compute, you want to get to N2P, you get a really bad process technology. For some of the HBM and HBM5s, we we decided that we'll have a different process technology and so be able to manage the cost and the power profile of going to HBM4 as well.

27:50 So I think that's really drives again uh our philosophy on hey, how do we make sure that we are more cost efficient over generation and more energy efficient over generation? And so that's that's where some of the along with the geometry, the decisions to go chiplet really helped us there.

28:09 Got it, got it. That makes a lot of sense. Okay, so you're you're talking about a lot of decisions that are very beneficial for decode, um, for the memory-bound part of the workload. Um, maybe talk to me a little bit about pre-fill. I mean, are within a rack of 96 accelerators, will you still ultimately disaggregate pre-fill and decode?

28:29 And it does that live within the rack? And and you you must have have enough compute where you feel like you know, with enough chips, you can still do pre-fill efficiently.

28:38 Absolutely. No, we did we did a lot of um modeling on understanding what our pre-fill to decode ratios are inside the fleet. And then um uh we are pretty confident is is you know, even with what we have sent right now, like what we have in our uh chips and and over the rack, we can absolutely deliver that pre-fill to decode ratios.

29:00 I think you bring up a good point, is what is the pre-field desegregation, like pre-fuel decode desegregation strategy. And and I think this is where kind of our strength in networking comes along because we have one scale-up fabric, right? And so we can absolutely take instances of pre-fill uh that have multiple GPUs and position them in one rack or spread them across rack, and then instances of decode in the same scale-up fabric, right?

29:26 Um, and so this really helps us because we don't really have to go over a scale-out network to actually pass the KV from a pre-fill to a decode position. We can do that all over the same scale. And so we we we do run our fleet with desegregation, and we are able to take some of those Mayas and run it as a as a pre-fill instance and some of them as decode.

29:47 And when we we kind of match rate them, so depending on how many pre-fill uh machines you have and what is the um token output on those machines, we try to match that with our decode side as well.

29:59 Nice, nice, got it. Okay, so yeah, let's go into networking topology. I know yours is two-tiered, but um tell us, you know, I think maybe zooming out, you know, people like to think about like where is it copper, where is it optics, how fast are the links, um, how many accelerators does it connect? And maybe even at a high level, like um if you can fit 96 accelerators in one rack and you're doing uh mixture of experts, inference at scale, like why do you even need to go beyond one rack?

30:28 Why need superpods?

30:29 Yeah, yeah. No, I think uh it's a great question. So let me kind of build it up from the bottom and then kind of talk about it. Uh at the at the lowest level of our network technology, we have uh our uh integrated NICs. Uh we introduced them in Maya 200. We continue to uh advance the capability of those NICs.

30:46 They are on-dye NICs, extremely power area and power efficient, and they're really built for scale up, right? So we're not trying to solve millions of endpoints, talking about a you know general cloud network NIC kind of thing. We're trying to basically look at it as hey, what does a scale-up domain look like?

31:03 It needs to be extremely high performance, right? It needs to be low latency, you know, we have we are going to continue to increase the bandwidth. That's kind of uh what uh a tensor parallelism or some form of you know workload that's spread across um uh nodes in the rack would need. And so looking at the communication patterns, we've optimized the now from there.

31:24 We have used standard Ethernet you know, switches and cables because we do believe, again, with the philosophy of maintaining the cost and and not trying to go into more exotic uh sometimes um uh implementations, they've stayed on Ethernet uh to really benefit from the scale economics of that industry.

31:45 And then we have built uh a node topology that keeps some of those links within the server itself, because again, we believe that um neighboring uh Mayas would have a lot more higher communication, especially if you're doing uh uh a tensor parallelism, so where you're taking those tensors and just spreading them across a few Mayas, a couple of Mayas in the node, or I think that's where a lot of the bandwidth stays within the node, and then half of that bandwidth goes to the switch links, right?

32:13 So that's the first tier of switch. Um, and then that's that 96 accelerators all have you know connections to the first tier of switch. And then we also decided that hey, what if we wanted to, you know, as the models grew bigger, or as we started to look at better and higher and higher sharding schemes, or we thought of you know flexible job placement so that the allocators job actually, the software allocators orchestration starts becoming a lot more easier.

32:43 Like we felt that a second tier of network would really help us scale uh into those use cases. And we have our own transport on top of Ethernet. So we were able to then use some of those transport features to be able to say, yeah, I can actually scale now and have a second tier, and so we can have accelerators from one rack talk to another accelerator from another rack using the same stack of communication, um, communication stack that's been developed for in-rack as well as cross-rack.

33:16 So that really helps us now get into this place where oh, you can start expanding the cluster size to about uh, let's say 128 racks in the same scale-up domain. You know that copper has a reach, right? And so we stayed with copper inside the rack. But as soon as we went from you know the the switch in the rack to the next level of switch, we started using optics.

33:41 And you'll start seeing some of those uh the systems uh come next year as well, where you'll start seeing you know more and more optics being used in the scale up domain, really to find that reach and to build these larger and larger uh pods or superpods.

33:56 Got it, got it. Okay, this is really interesting. So you said at first you don't really need to design around scale out because you're not trying to do a million accelerators for a training job. You're just focused on inference, as we talked about before. So you're really focused on scale up. You said uh the the four there's like four accelerators that are uh like tightly coupled to each other, like directly connected, and and you're trying to think about splitting up the workload so that neighbors talk to each other a lot uh you know, directly over copper.

34:25 All the accelerators in the rack are connected via uh the first level of switch to each other, also copper. And then the switches connect to other switches like rack to rack um using optics. Using optics. Awesome, interesting. And it sounds very you know powerful and flexible. Um, do you do you have any two cents?

34:44 Are you guys saying anything about how Microsoft feels on uh near package optics versus CPO, NPO versus CPO?

34:52 Yeah, no, I think I think uh the story is developing. Uh we do we do believe that uh optics in scale up is something that uh everybody will eventually move towards. Um at the same time, I think we need to bring the reliability and the cost of optics along with the right supply chain uh you know motion to make sure that the industry kind of everybody moves together in some ways.

35:18 And I think uh in terms of NPO and CPO, there are real trade-offs there. Consider again, talking about you know what do we do with reliability and how do we make sure that uh with NPO you have a less disruptive, uh, you know, much I'd say streamlined path into the server and how you design it. Uh but over time, we also know that CPO gives you the power efficiency at extremely high bandwidth rates.

35:45 And so I think the industry is gonna kind of start with a little bit of an NPO, trying to get a feel of how do we manufacture this, how do we make this very serviceable, how do we improve the reliability and the cost, and then move towards a more integrated approach, which is CPO on the XPU side. And similarly on the Switch side, we are already starting to see some of those motions as well.

36:08 Yeah, yeah, makes a lot of sense. So then talk to me at the rack scale. Let's talk about like power and cooling. I saw a sweet picture with this big thing sitting next to a rack of the the 96 accelerators. Uh what was going on there? And um, when you're talking about cooling, like are you thinking about deploying this only in places that are already support liquid cooling?

36:32 Or can you go into air-cooled data centers and you know why why did you make these cooling decisions?

36:39 Yeah, no, this is again uh really what I think is is the core design aspect of where we play out really well, right? I mean, we have designed Maya to be deployed in both an air-cooled and a liquid-cooled data center. So we know that you know, as we expand this to you know global and planet scale, there are going to be opportunities to say, hey, there's a there's a data center here and and it needs AI infrastructure.

37:06 And sometimes you are not have access to liquid-cooled facilities because they are still being developed. And so um, this is a great opportunistic deployment for us to say, hey, we can actually pair a Maya with a heat exchanger unit. Again, going to the power of the energy efficiency of the rack at 167 kilowatts, we are able to actually design what we call a heat exchanger unit.

37:32 Uh, and and it's really what it is. It takes the the Maya chip itself is liquid-cooled, right, along with other components inside the server. But then when the warm water comes in, it can be put next to a rack that has fans in it and it cools down. It takes the ambient uh temperature of the air-cooled data center, takes the air, flows it through this liquid, and brings it back at a lower temperature so that Maya systems can run at the full 167 kilowatt rack capability.

37:59 So we have we have designed this heat exchanger unit with Maya, with our data center deployment in mind, and are able to then look at it and say, if I need to scale this to different parts of the world where air cool data centers are still available, I can easily scale it there. At the same time, if it's a natively facility water-cooled data center, uh, you don't need to use the heat exchanger unit.

38:22 You can just take the Maya Rack and then connect it directly to the plumbing that's available in the data center and be able to just deploy it in a next generation liquid cool data center. So that really gives us again the deployment flexibility and and and it has to be architected up front. This is not a decision you can make once the server everything is built.

38:42 You have to architect these decisions up front because this is how your your deployment and data center operations and where you deploy these at scale all have to kind of play in at the level that we are deploying these systems at.

38:59 Sure. Yeah, that's really cool. I like that you keep the optionality to be able to deploy these wherever you want. And if it can't support liquid cooling like uh normally, then you have this heat exchange unit to make it happen. So but that makes me wonder then, like where how do you guys decide where to deploy these?

39:21 I mean, are you putting them in the same place where you've got the Maya 200s? Are you trying to put them near for inference, like near densely populated places to have the inference as close as possible? Or like, like how are these decisions in the world?

39:35 But but uh but I think uh going back to you know, the kind of starting with uh we are seeing our customers use a lot of our Microsoft products and services. And so we talk to our internal product teams and services and understand where the demand is and where the growth is and where the capacity needs to land.

39:53 And then based on you know the demand and growth, uh we we have a pretty well-oiled process inside the company to make sure that the systems are ready for those data centers. And um, at the same time, uh we want to make sure that uh these systems are delivering that class of service that that this thing wants.

40:16 So we do a lot of work internally to make sure that those services are first kind of in some ways they run uh let's call it shadow traffic on it, just make sure everything is is correct, everything is you know hitting the the right customer experiences and expectations. And when the services are ready to scale, you have them deployed in the regions that they want.

40:39 That makes sense. So um, okay, so when you're talking about like your internal customers, how much are they using Maya 200 today? And then uh as far as Maya 300 goes, like from a software perspective, you you talked about like the software-defined data flow architecture. Like, is it going to be a big lift for those internal customers to move to the next version of Maya?

41:04 No, absolutely not. In fact, uh I think what this is, is it's really at the kernel level, right? Where you all these optimizations play in. And uh we have a very robust like you know, programming stack that's already ready so we can build uh the right kernels. But I think what I'm really, really excited about and what has changed since maybe the last time we talked is the ability to actually program these machines really well and extract uh almost the peak performance out of this machine.

41:35 And that's because of you know how we can now take agents in the loop and use agentic kernel optimization to build and understand, put the right optimizations in place even before the customer or the application sees it, right? So for them, this is mostly an endpoint, an API endpoint kind of a thing where they don't have to worry about the underlying architecture.

42:01 That's kind of abstracted away from them. And all of this uh you know, data flow architecture, the placement of the data, how do we uh continue to increase the performance of those specific kernels? That's all hidden from them, right? That's part of the library that gets uh built along with these um systems.

42:20 Gotcha, gotcha. So it's not a problem for your customers in as far as uh you know, internally needing to upgrade the kernels because you know that those specific kernels are just the things running over and over and over and over and doing all the math. Um, you're saying that uh agencai coding capabilities have progressed so quickly, you know, in the last six to nine months that um you can also throw a swarm of agents at it and that helps you uh even more quickly optimize these kernels.

42:50 Exactly.

42:50 So day zero model availability is not gonna be a problem. I think you're gonna be able to, like you said, really farm these out. If there are new models or new kernels that need to be written, um these these agents are able to continue to run in a loop and able to create new versions of optimized kernels that we can then populate back into our kernel libraries that can then expose to just LLM calls?

43:19 That's amazing. Uh so as you stand these up to serve internal customers, essentially, you know, tokens as a service, do you ultimately uh configure them differently to meet different like SLAs? Like, does someone want, you know, five milliseconds of latency, someone else is okay with 20 milliseconds, but they want more throughput or cheaper tokens?

43:41 Or or how how are these getting configured?

43:44 Yeah, and I think it really the answer honestly really depends. Um like you said uh and you mentioned it that there's kind of an interactive versus throughput charts that people look at. And and what when you actually look at the charts, they are different points of chart. And different points of optimization of the same system.

44:03 And so depending on where your workload is leaning towards, like if you have chat or reasoning heavy, you will have a certain kind of aspect of what the workload is stressing. Then you also want to understand what your interactivity is. Is it like, hey, I just finished a team's meeting and I just want to summarize what it is?

44:24 Well, fine, we could just batch that off and make it a throughput equation. Whereas this is a super critical agency loop. It has extremely low latency requirements. Yes, you have to reduce the concurrency and batching of what you can do in that system so that most of the resources are given to that particular task.

44:49 Now there is model infrastructure co-design that can definitely happen, right? So when we look at, for example, working with our own model teams, we can expose the strengths of the system to the models team. And what they do is now they say, oh great, now I know what the systems are capable of. I can write some of the model infra decisions can be made in a better way.

45:13 And so I think that can also help continue to extract more and more out of the system. But I would say there are like those two layers of optimization you can do. You know, how much are you really looking at model infra core design? And then how are you looking at the task that's coming to you and then finding the right optimal configuration and sharding scheme and the right interactivity point, or again, going back to what is that class of service and capability versus price that I have to deliver for this particular request?

45:43 And how do I do that in these systems? So that's really in some ways you can optimize this, optimize, make that uh optimization decision at runtime, think of it that way.

45:54 Sure, sure. Okay, so zooming way out, last question, we've covered so much ground here. This has been awesome. Um, how should we think when we look at Microsoft and the Maya 300, Microsoft Azure, how should we ultimately think about the benefits to Azure? Is it that uh Azure can now run internal workloads more cost effectively?

46:17 Um, is it, oh, here's a vector where actually maybe there's an opportunity to rent these out or sell them to other customers? Or like, yeah, how should we think about the big picture?

46:30 Yeah, I think the big picture is how do how do our customers really benefit from the investments we are making? I think that's the real the real story here. And like I said, it's it's not about the chip, it's not about Maya itself, but building an AI inference system that really maximizes the token throughput or the intelligence you can create out of a given power envelope or a given cost envelope or a given, you know, even a capacity constraint.

46:57 And so I do believe that with with Maya 300, we've had this significant leap in architectural performance where we can deliver these services. You know, the tokens can be, you can generate new business models, so you can create new feature set at a different cost point and capability points. And that's what really we are trying to see is how do we make sure that we can now diffuse this intelligence through not just frontier models, but frontier infrastructure.

47:27 I think that's that's the play here. We have to build the infrastructure that can serve our customers. And and Satya mentioned this when we can actually drive down the cost point of delivering these, the usage and the consumption and the productivity, it should show up in productivity stats and GDP stats across the world when more and more people are able to take some of these AI features and they are scalable, they are much more efficient, they are much more affordable, they are much more sustainable from a data center perspective.

47:57 And so all of that kind of uh crews towards how the world is going to use AI in in their day-to-day lives. Um, so I hope that really frontier, as we talk about frontier models, it's really the frontier infrastructure that also kind of dives into diffusing AI.

48:17 Yeah, definitely. Okay, so to summarize, it sounds like you're saying, hey, the world has a fixed amount of power and a fixed amount of dollars, and we're just trying to help customers get more intelligence out of their budget, out of their power budget, out of their financial budget. And the Maya 300 is a step in the right direction, co-designed with uh thinking about minimizing data movement and um co-designed from uh accelerator to memory trade-off decisions to network to cooling and racks, all the way co-designed to just it's for the customer trying to do more for them, essentially.

48:55 Exactly.

48:56 All right, well, I'll be a Microsoft salesperson for you. There you go.

49:00 You guys are thank you, Austin. It was such a pleasure.

49:03 Okay, yeah, it's always a pleasure. And you know, let me know when you're ready to talk about the Maya 400.

49:07 That'll be soon too.

49:10 Okay, excellent. All right, thank you. Thank you.