Patreon Preview: The Harms of Generative AI w/ Alex Hanna

0:00 Hey, this is Paris. I hope you enjoyed the Data Vampire series that we did back in October. It's had a fantastic response. And for that series, I spoke to a bunch of experts, and now we're releasing the full-length versions of those interviews for our supporters over on Patreon.com. And I wanted to give you a preview of what those interviews sound like.

0:19 So you can consider whether to go to patreon.com slash techwon'save us, become a supporter yourself, so you can learn even more about the important topics that we dug into in that special series. So enjoy this clip from my interview with Alex Hanna. Why are these generative AI models so computationally intensive?

0:38 Yeah, so they're computationally intensive because they are so large and the process of training is a pretty computationally intensive process. So it depends on how far you want to go back. But if you go back to the original innovation of neural networks and the kind of advent of backpropagation, backpropagation is just a really intensive process because you have this architecture.

1:15 There's a few that are more or less popular. It depends on, you know, the latest and greatest large language model, you know, has a certain kind of architecture, you know, and it it differs. You know, like I can't imagine there's a lot of daylight between the cutting-edge models at OpenAI versus anthropic versus Google.

1:35 Um, maybe there's some kind of differentiations that they have, but at the end of the day, the actual parameter fitting, the actual training process requires this this goes through this process of backpropagation to actually uh have the weights on all the neurons. And so the um yeah, so in that case, backprop is just basically huge matrix multiplication operations, and this is why GPUs have taken off as a sort of specialized hardware that works really well for this.

2:09 So NVIDIA, who you know been making these things primarily for graphics, then you know, when AI has taken off, now NVIDIA has you know this huge market cap, although it's going down now as the the bubble is starting to deflate. And so NVIDIA, you know, has just been making so much bang, selling these GBUs for training.

2:30 So you're just doing massive distributed matrix multiplication across you know hundreds of thousands of these of these H100 chips. And um, so that just makes it incredibly intensive computationally and with power. That's the main thing that it's attributed to, to training. But then all there's also the kind of power that it takes when it comes to inference and actually doing prediction.

2:53 I mean, you still have to you still have to draw upon all those those weights and then do all the multiplication of you know whatever the data input is. Marginally, that is a lot less, but over you know, cumulatively, that takes a shit ton of computation and power. Those are the two dimensions. We talk a lot about training, but the inference, you know, just like getting a bunch of users, is uh also intensive computationally and resource-wise.

3:20 Yeah, so I guess the focus was initially, especially as these things were rolling out, much more on the training and you know, how much went into kind of creating these models in the first place. But as the tools have proliferated, whether it is from OpenAI popularizing Chat GPT or Google like sticking Gemini in all these different applications, then there's a lot of like computationally intensive tasks that come from that as well, especially because you have so many people using these things.

3:50 Yeah, that's right. And you know, Google had been very optimistic about this on the outset. I mean, there had been the kind of paper that Google had written that had David Patterson at the lead as the lead author, and it had some kind of ridiculous title that was like, you know, the you know, the cost of language, large language models will increase and plateau or something of that nature.

4:12 And they really had this kind of vision. I mean, it it made a lot of very strong assumptions in that paper. I mean, one of them being that you could kind of optimize for training these things where you know you had almost all green energy, or you could train them in off hours. But the huge error there was the focus on the marginal cost of training, you know, maybe a single or a class of models and then doing kind of inference rather than seeing that these things are just popping up everywhere and training is what's gonna happen.

4:48 The training cost is going to be the huge cost from the beginning and it's going to put the most stress on the power grid. But if you have these things continuing to do inference, and if you want to actually grow your user base, then yeah, you're gonna have a hundred different models doing inference.

5:05 And you know, if you're open AI and you say that, you know, I don't know how many active users they have right now, they brag that they had a hundred million active users a month after Chat GPT came out, even though I'm sure that's precipitously dropped, you know, because people are like, okay, this thing was fun to play with, and now I don't give a shit.

5:24 But, you know, if you still have kind of a class of models, especially as they're trying to do a big push for enterprise sales and they're being foisted on people in ways that where they can't opt out, then yeah, I mean, then you're not going to get that that marginal cost is going to really add up.

5:40 If you found that clip intriguing, go to patreon.com slash tech won't save us, where you can become a supporter, support the work that goes into making the show, and get the full length version of that interview. Thanks so much.