Generative AI is Not Inevitable w/ Emily M. Bender and Alex Hanna

0:00 There's this inevitability narrative that we hear from the tech companies all the time, and that is a bid to steal our agency. And so, you know, Alex said it can be inspiring to take a principled stand, and one part of that is to say, no, I don't believe you. If you say, well, this is here, we have to learn to live with it.

0:14 No.

0:15unknown Right?

0:15 This there's nothing inevitable either in the development of something called AI or AGI or in the sort of continued presence of this tech in our lives.

0:41 I'm your host, Paris Marks, and this week I have two fantastic guests joining me on the show. Alex Hanna is Director of Research at the Distributed AI Institute, and Emily M. Bender is a professor in the Department of Linguistics at the University of Washington. They have a new book out called The AI Con: How to Fight Big Tech's Hype and Create the Future We Want.

0:59 Sounds like exactly the type of thing that we like to talk about on this show. Alex and Emily have been on the show in the past, of course. You know, Alex for the series that we did called Data Vampires, Looking at Data Centers and things like that, and Emily to talk to us about AI and her input on that as a computational linguist.

1:16 Their new book is a great primer for people who are looking to understand what is actually going on with the AI hype that we've been seeing over the past few years, all of the problems with it, and what the actual goals of these people are. And importantly, they also look at how we can push back against these things, right?

1:33 Both as individuals, but also in collective ways through government and through other means. And I wanted to make that a big part of our discussion, but we started off by talking about, you know, some of the bigger ideas of the book around AI hype, but also a number of the things that we have been seeing occurring at the moment that I wanted to get their input on as people who have been paying such close attention to everything going on with AI, not just for the past couple of years, but even before the hype around Chat GPT and generative AI and the types of things that have been motivating the tech industry and investment since 2022.

2:06 I have to be honest, I really don't think that there is much more context that I need to give you for this episode. It's just a really great conversation about where we stand right now with AI and how we might want to push back against the things that we're seeing in order to limit the harms that we're seeing come from it, but also the broader futures that these tech billionaires are trying to bring into being.

2:25 So, with that said, if you do enjoy this conversation, make sure to leave a five-star review on your podcast platform of choice. You can share the show on social media or with any friends or colleagues who you think would learn from it. And if you do want to support the work that goes into making tech won't save us every single week, so we can keep having these critical, in-depth conversations on important issues, not just affecting tech, but affecting all of us.

2:45 You can join supporters like Maddie from Toronto, Alex from Edmonton, Vladimir from Brazil, Brian from Houston, Abe from Grand Rapids, Violet from Pennsylvania, Matt from Maryland, and Jan from Prague by going to patreon.com slash TechWon't Save Us, where you can become a supporter as well. Thanks so much and enjoy this week's conversation.

3:02 Emily, welcome back to Tech Won't Save Us.

3:05 Delighted to be here again. Thank you.

3:06 And Alex, it's great to have you on the main show, not just uh, you know, one of our series as you were last time you were on. Thanks for having us, Ferris. Absolutely. You know, I really enjoyed the new book, The AI Con. I feel like if you have someone in your life who is like, what do I need to know about AI?

3:20 Like, what is the problem with this? This is like the perfect book to be like, pick this up, read it. It has all the things you need to know in there. It's really concise, but also still has all the detail that you need to actually understand what is going on with this industry and why you should be concerned about this products and what these CEOs and things are actually doing here.

3:40 So I really applaud you on the book. And so my first question was really like, how did you decide that you wanted to put this together? How did you decide we need to write this book about AI to really start to dispel these myths that uh that are so persistent?

3:53 Oh, I'd say it actually starts with our podcast. So we run Mystery AI Hype Theater 3000, which um is an accidental podcast. If you listen to the first couple episodes, you'll tell that we did not know we were making a podcast. And you know, we were doing the work in that podcast of trying to top the bubble of AI hype.

4:10 And it's a lot of fun. We still do it. It's roughly twice a month, and we do different topics each time. But it started feeling like we could reach a larger audience if we could construct the argument and put it in one sort of neat package.

4:21 Yeah. And so we were kicking the idea of a book around and got approached by an agent. We're thinking, well, yeah, we could really think about a way to tell a very coherent narrative. And I think one of the things that we appreciate you naming that it's pretty accessible, because the idea is let's start kind of from zero or start from a place of I've heard about this thing called AI, and really explaining what these technologies are, explaining that AI is not one thing, um, and trying to demystify everything that it just has sparkles in front of it.

4:59 Yeah, I I was really mad when they started stealing the sparkle emoji and making it seem like it was AI.

5:05 I know. It's a fun emoji, right? It's a great emoji. And whenever I see something that has it, now I have like a, you know, a moment, it's like a jump scare. I'm like, oh no, but I want it to be good.

5:15 Absolutely. So the book, I I feel like one of the big focuses in the book is AI hype, right? And you know, the the effect that AI hype has on our understanding of this, how it's able to be sold to the wider public in a way that hides some of the harms that come of this technology. So why focus on that so much?

5:33 And and how do you assess the state of AI hype in May of 2025 when we're speaking and when your book is coming out?

5:39 When we finished the manuscript in September of 2024, there was this like urgency. It's like we really wish that we could have had the book on the shelves right when we finished writing it. I'm sure all authors do, like, I'm done, how come nobody can read it? Right. And I remember saying, it will be a better world if this book is not needed in May of 2025.

5:57 And unfortunately, we are not in that better world, right? The hype continues. And why hype? I think part of it is that it really draws on our expertise. So I'm a linguist, Alex is a sociologist, and we can look at how people talk about things, how they sell things, how the large language models play on our linguistic competence as people and sort of open up a possibility for hype.

6:20 Sociologically, what does that do in the world? I think it gives us an interesting angle in that we don't see represented so much in there's other fantastic books that people should be reading about the AI industry and the harms that are being done. But I think hype was an open space we could move into.

6:34 Yeah, I would say that one of the kind of handles of hype from a sociological perspective is that it gives you a sense of talk about the political economy of the AI industry. It gives you a sense of talking about, well, these are the promises of the industry, and here are the institutions that are surrounding it, as well as the massive amounts of investment going into it, right?

6:56 It's kind of fascinating too here in May in 2025. I had just re-skeeted or quote-skeeted this study that IBM had done of CEOs, and there was some really flooring statistics in that study. The one thing I posted to Blue Sky on it, I mean, there's there's many different parts of it which are flooring, but the one that I quote-skeeted was surveyed CEOs report only 25% of AI initiatives have delivered expected return on investment over the last few years, and only 16% have scaled enterprise-wide.

7:29 So, and then the really flooring part of that, if you go further down, is basically I think something like around 68% of those firms still plan to go full steam ahead. So we're still within the era of AI hype. And unfortunately, or maybe fortunately for us, unfortunately for the world, the book is still very relevant.

7:50 It's always wild to see those stats, right? Like it's pretty clear it's not paying off, but we're still going to keep pursuing these sorts of things. I remember there was one last year where they had surveyed a bunch of employees and a bunch of like management or like C-suite people or whatever. And the employees overwhelmingly were like, this makes me do more work.

8:08 Like it's not very efficient, it's not working very well. And like most of the managers were like, it's great that we're implementing these things. It's very effective. Like there was this like total disconnect there, right? But I I want to double down on that second part of my question because I feel like there have been multiple moments, even like you're talking about, you know, end of last summer and whatnot, where we were talking about, oh, is the the bubble finally going to burst?

8:31 Is the hype finally exhausted? Are we finally waking up to what is going on here and you know it's going to start collapsing? And we'll see what kind of comes on the other side of it. But it feels like again and again it has managed to continue limping on. So I wonder how you assess the moment that we're in right now in terms of AI, the support that exists for it, how it is continuing to maintain that interest and that investment.

8:57 Yeah, I mean, it's it's interesting because I think what it is is it's it kind of signals that this is a bubble, not a bubble-shaped thing, you know, and it makes me think about well, what's going to happen here and what happens after AI. And, you know, we spent a little bit of time at the end of the book thinking about when the bubble bursts, where however it bursts, because it might burst slow, it might burst fast, it might have these people, you know, tossing the good money after bad, which then becomes bad money.

9:28 Um, and the way in which, you know, what is left behind in that. And that's the kind of data center growth and energy and waste that's been generated by that, and the carbon that's put into the atmosphere, as well as the air, other air pollution and small particulates. There's the jobs that have been completely decimated, the fields that have been decimated.

9:49 The thing is, like, I don't really know. I mean, it's hard to sit here from our chairs and try to play, you know, industry analysts, but it's like, what kind of shape does this look like? And I mean, I think there's either probably going to be one of two ways, maybe one of four ways. I'm not sure what the second axis is.

10:09 It's either the Uber model, which is, well, we're just going to keep on tossing money at this, and at least one day this is going to turn an investment, and that's after a lot of capital has been destroyed. But there's going to be some kind of sustainable path where it's, I don't know, you have a few data centers and that's going to help students cheat on homework or whatever.

10:30 Or it'll be the big explosive kind of way, which may be less plausible, but maybe there's something that really disrupts it and there's a real emperor has no clothes sort of moment. But it's hard to speculate at this point.

10:43 And you ask, like, why is it still limping along? Why was last summer not the end of the bubble? And I'm, again, not an industry analyst, but from where I sit, it looks like it's a question of just how much money is there already, and maybe something of a sunk cost fallacy on the part of the venture capitalists.

10:59 And the promise, and it comes back to the hype, right? The hype says if we can do this thing, if we can actually build the thing we say we're going to build, it is going to be infinitely valuable. And so somebody whose job it is to maximize profit who believes that hype, like, why would they not invest, I guess?

11:16 No, it's it's a really good point. And Alex, you know, you mentioned what happens after the bubble burst. Uh, I want to come back to that a little bit later while we have a discussion about some of the stuff you discussed near the end of the book, but there are a few other things I want to dig into before that.

11:29 Obviously, this boosterism is very much closely associated with the hype that you're talking about here, right? And I feel like uh again and again, unfortunately, we see this boosterism not just coming from people at the top of these companies and venture capital firms who you would expect to be boosters of this technology, right?

11:47 They're hoping to make a lot of money from it and to kind of spread it into the world. But we also often see that from a lot of media, right? And and a lot of reporting on this. What role does that play in helping to maintain this idea that AI is the next big thing, that it is transforming the world in the way that some of these tech folks want us to believe?

12:06 And how does it distort the conversation and the public understanding?

12:09 Yeah, I think every time a journalist uses the phrase artificial intelligence uncritically, right? They say, oh, there's a new AI-powered tool that connects YZ, that misinforms the public. And I think about, for example, teachers who are now faced with lots of pressure to bring AI into the classroom so that their students are trained for the workplace of the future, right?

12:27 As if that's what education is about. And that's another sad topic. But I think that if we had more critical journalism, and there is, you know, wonderful journalism, I'm constantly pointing people to your podcast, Parrots, and really appreciate the work at 404 Media and people like Karen Howe, whose fantastic book is coming out.

12:44 So there is wonderful critical tech journalism, but there's also a lot of credulous access journalism. And one of the things that I hope we can do with this book is help people sort of navigate to the critical journalism and know what they're looking for so that they can continue to get good information from the people who are asking the tough questions.

13:02 Yeah, and I mean the the credulity comes a little bit, and I'm I'd be interested in a bit of a history of tech journalism, but so much of tech journalism has been all about access or has been all about just, ooh, look at this fun whiz-bang thing. And so you have, you know, so unfortunately, you know, you have your folks like Kira Switcher who was like, Elon Musk was great until he lost his mind.

13:25 I'm like, Yeah, he was great until he called me an asshole, and now he's not great and stopped returning my calls. Exactly. Yeah, like that man lost his mind a while ago, or you know, was died in the wool, someone that you shouldn't trust. And you've had a number of other folks, and we talked, I think, before we got into call about Kevin Roos and Casey Newton.

13:44 And Kevin Roos has been this person that I think has really been brought in to the kind of uh he's been kind of dumbfounded by all these different kinds of the outputs of chatbots, you know. I think there was that one point where you had this conversation with what was the Bing chatbot, Sydney. And then at some point it had he had said it had tried to get him to leave his wife, and I don't think he's recovered from that because I think it's everything from that has really been kind of flooring.

14:11 And I mean, it's been this kind of thing that it's just granting the tech billionaires so much more grace than you would grant any kind of area of research that you didn't know anything about. Like, how would you explain this? I mean, there's a lot of hand waving that gets going around. And you know, we saw this interview with a CBS show they did with Demas Hasabis Hasebis.

14:34 And the way I was reading the transcript, and they started off with, you know, you know, he grew up and he was a chess prodigy at 12, and you know, and then they had this interview where he, you know, said it was going to cure cancer and cure all illness in 10 years. And you can't let people just get away with saying this complete batshit like that, right?

14:54 Yeah, it's just like, wow, oh my god, the future is going to be so incredible. You don't have to follow through on it, but it sounds so great, right? But no, I I think that those are all really important points, right? Because the public really relies on media in many senses. Obviously, we've seen this erosion in trust with some media over the past number of years, but to a large degree, people do still look to media and journalism to get an understanding of what is going on in the world.

15:19 And when they approach this and get these takes on AI and technology that are overwhelmingly booster-ish, that are overwhelmingly repeating the press statements of these tech companies, that doesn't actually help the public understand what is happening, but just tells them what the companies want them to believe.

15:36 And that's not how you have like an informed discussion as you're trying to promote and encourage us to have in this book, right? I feel like another piece of this, right? We look at the journalism piece, but another piece is how these discourses play out within the tech industry themselves. And in the book, you talk about the doomers and the boosters.

15:52 We've talked about the boosters, but you know, there's these doomers as well who promote this notion that AI is going to be so powerful and is a big threat to us. Can you talk to us about how these narratives that in some senses might seem to be in conflict with one another actually work together to promote this notion that AI is huge and powerful and going to change our lives in all these ways?

16:14 So, in a nutshell, the AI boosters say AI is a thing, it is imminent, and it is going to be all-powerful and solve all of our problems. And the AI doomers say AI is a thing, it is imminent, it's going to be all-powerful, and it's going to kill us all. And saying it that way, you can see very quickly that it's actually the same position.

16:32 And it's just what turn do you take at the end? And the really frustrating thing is that the booster versus doomer discourse sets it up as these are the full range of possibilities. You're at one end, the other, or somewhere in between, that is it. And coming back to journalism, a lot of the journalism sets it up that way too.

16:49 And what we say, I have been saying for a long time and say in the book is no, that's actually two sides of the same coin. It is one small part of the range of possible ways of seeing things, and it is not a reality-based part of it.

16:59 Yeah. And so we talk in the book a lot about these different, I mean, we don't we don't go into the different kind of flavors of Doomerism. We kind of gesture to the test real bundle, the transhumanists, extropionists, singularity, cosmism. Okay, now we're to the real rationalism, effective altruism, and long termism.

17:17 I mean, you've had to meet on the on the program a few times and respect for having to memorize so well. Yeah, it's it's hard, it doesn't exactly roll off the tongue when you actually spell out the acronym.

17:28 I finally figured out what extropeism means, actually. From Adam Beckett's book, More Everything Forever. It's extropy as the opposite of entropy.

17:37 Oh, I'm learning things every day here. I I got Adam's book on my big pile of books too. And and so, you know, we talk about getting into that and you know the different flavors and sort of, you know, the kind of thing that's interesting is that you have Yudowski as kind of this person who is like kind of this true believer, this person that is somebody on Blue Sky called him Big Ud, and I thought that was kind of great.

18:00 Um, so you have this kind of towering figure, the people who are like really getting high on their own supply. Then you have people who also have the kind of mixed incentives, you know, like Elon Musk, who's, you know, a long-termist, says he's a long-termist, but also, you know, is is of course just trying to make money and you know, has status and and power aspirations too.

18:20 So there's kind of a mix, you know, it's all the same effect, kind of at the end, whether you're kind of a strong long-termist or a weak long-termist, but it does lead to this kind of doomerism and underlines the same bottom line that AI is inevitable. It's a thing we need to believe in. It's a thing that we should put our trust in.

18:37 And it's just a matter of aligning it correctly.

18:39 And I feel like one of the ways that this becomes really concerning, which of course you you wrote about in the book, is when you see this kind of booster, doomer dichotomy or relationship, then be the one that defines discussions in policymaking, right? Whether it's in the US government or or other parts of the world, where all of a sudden that is the frame through which the conversation is happening.

19:01 And, you know, like you discuss, it very effectively shapes that conversation in a favorable way, right?

19:07 Yeah, 100%. I mean, we saw this. We started the book with an anecdote about the Schumer's Insight Forum, and especially focusing on the ninth one, in which he asks, you know, what the what is the P doom and what is the P hope? P doom is the probability of doom, and P hope is the probability of hope. So P parentheses, doom, and parentheses.

19:30 I'm sure this is language that Chuck Schumer is using all the time in his everyday conversations.

19:35 Yeah, Chuck Schumer is he's just walking down the halls of the the Dirksen office building in DC and asking, hey bro, what's your P Doom? And so, you know, this this got some reporting to there was like very credulous reporting in like the SF standard or whatever, saying the hottest question around Silicon Valley is like, what is your P Doom?

19:57 And oh my god. Yeah, but I mean, anyways, on the policy element, I mean, that became a frame through which to have conversations during that insight forum. And then the output of all those insight forums were that the whole thing focused on innovation or national security and whatnot, and had very little bit to say about present-day harms.

20:17 Yeah, it's a real concern, right? And and when you think about present-day harms, I feel like this is something that we are increasingly talking about. It's something that all of us talk about a lot when it comes to AI, but I feel like it's slowly making its way more into the general conversation. And I wanted to pick up on a story that was published just recently in New York magazine talking about how generative AI is increasingly kind of moving into education, you know, being used by students and educators and even administration.

20:42 And I wanted to ask, because you've both been involved in these spaces for a while, what you make of how generative AI has moved into education in the past couple of years and and the effects that it has had there.

20:54 So I got to speak at the Pac-12 Deans Conference last spring, back when there was a Pac-12. Um, so this is uh deans from universities up and down the West Coast. I think we not just the US, we had some folks from Canada. And I was brought in to talk about AI. And I got to say to these deans, look, the reason this is your problem now is not that there's been some enormous breakthrough in the technology, but rather there's a lot of money and people are trying to push it on you because they want to start recouping some of their investments.

21:22 Like that's that's all this is. And the second thing I said to them is the only use of something like ChatGPT at a university is as a contrast die test to understand where more resources are needed. When a student is turned to ChatGPT, it is either the instructor has not motivated the importance of doing the homework, and so like why bother?

21:42 Or that student doesn't have enough resources. They haven't had enough background to do the assignment, they don't have enough time because they're also working full time or whatever it is. But it's basically and if you got staff turning to Chat GPT, again, there's some lack of resources there, and that is sort of the only useful use case from a university's point of view, in my mind.

22:00 We talk a lot to K-12 educators who see we had a wonderful episode of our podcast with Adrienne Williams on talking about ed tech and you know she has a long history in the classroom watching how basically this stuff is just getting pushed on students and the students of the wealthiest families in the US, at least, I'm not sure what's going up in in Canada, are in schools with no screens in sight, right?

22:23 People who really have resources, who want the best, as we all do for our kids, are staring far clear of this. And just in general, I think that any time someone is using a chatbot in the context of any educational assignment, it's a missed learning opportunity.

22:38 I mean, universities have really fallen flat in the response of this. Either they have made deals with the companies, or they have I think they mentioned in that New York magazine article, like, you know, they they've sort of said, okay, maybe not, but like you decide, but they haven't really provided any strong guidance for for faculty.

23:00 And it's been sort of a missed opportunity. And so in the worst cases of it, it's it's kind of like, well, you as a professor should basically just throw up your hands and declare bankruptcy, and you just have to grade this as if someone had written it. And that is kind of the worst thing to do because now you're putting the onus on the professor to try to think about this and try to think about what it is in the context of their pedagogy.

23:29 There's also been turns to become just intensely carceral and just turn all the structure to the cops and saying that you need to police this really heavily. And both of those are terrible strategies. And I think that there really needs to be strong leadership from you know universities, but in where universities haven't pushed back, some leadership has come from some education unions where they're like, okay, this is where you can we can support our members and support a really supportive learning environment.

23:59 And so, for instance, in California here, Cal State, the Cal State system had one of the largest contracts. So Cal State has, I think, something like half a million students across all their campuses, the largest system in the US. And they ended up with a $16 million deal that they had with OpenAI for just a site license for a year, not even beyond that.

24:21 So huge waste of money. Cal State, like every system, in a budget crisis. And the California Faculty Association said, okay, slow down. No, we're not actually going to have this. We need to push back because we're not, why are we giving all our money? Why aren't we reinvesting in kind of pedagogy and thinking about what it would mean to really invest in our instructors and our students?

24:43 I want to add another worst-case scenario, which we've seen reported a few times, is especially in hybrid classrooms like I teach, where you've got some students in person and some online, we've seen a couple of universities proudly announcing that they have some AI students joining the class.

24:56 Oh, Jesus.

24:57 And the reporting on that is usually appalling, right? Sort of talking about them as students instead of text processing surveillance machines being added to classes. Other cases where you get faculty sort of proudly training chatbots on their own works and then putting those chatbots up as if they could be teaching assistance for the students.

25:15 Yeah, lots of terrible ideas.

25:18 Yeah, really. I hadn't even heard of those ones. That's yeah, that's terrible. When I was reading that story, and as I've been seeing stories about this come out more and more, like one of my concerns is really around critical thinking, right? And what this actually does to the broader society. And it's also not to say that this is a brand new thing with AI or generative AI, but it feels more like the continuation of a process of reducing education, reducing critical thought within society more broadly.

25:48 And this feels like it comes after like a long list of developments over many decades that have further kind of degraded that. And it just seems like a very kind of scary indication of where things are going.

26:00 Yeah, for sure. I mean, as a sociologist, my tendency in this book was to historicize. And so just taking the adage of many social sciences, always be historicizing. And so the onslaught of LLMs in the classroom has really been part of this process of the increasing casualization of the university, the increasing digitization of the university, the increasing surveillance nature of the university.

26:28 So, in the cases where the university, you know, at one point, at least in the US, where we have this private public system where whereas, you know, I remember speaking to someone at a at a student power conference once, and they're from Montreal. And uh, we just explained the notion of a private university to them.

26:49 And they were like, someone could just start a university and it just, you know, it is a thing that boggles the the Quebec mind.

26:57 Yeah, in Canada generally, like the notion of private universities are pretty, I don't think there are many up here at all. Like it's mainly public, yeah.

27:05 Yeah, yeah. It's it's some it's it's it's completely outside of what is what is considered to be kind of a legible institution. And so in this place where you know you had private investment universities where many flagship universities now have 8% or less investment in funds actually coming from the state where most of the funds are coming from donors or tuition or XYZ other other place.

27:31 And that's been happening for years, for decades. And we have, I think, a stat in the book. I don't know if it's in the main text or in the body of the book or in the footnotes where it talks about the decrease in tenure, the increase in casual labor instructors and lecturers and adjuncts, the rise in debt financing and debt servicing as being a huge part of expenditure.

27:54 I mean, this is the next line of the casualization of the university, in which universities are always telling their instructors and department heads to do more with less. And so it's not really surprising that this is the next point. I mean, the the last kind of thing we had was MOOCs. Everybody's gonna be on Coursera or whatever thing, or whatever.

28:16 You know, Andrew Ng is gonna replace your machine learning 101 class. But I mean, it's the next line of this. Now everybody's gonna have a personalized tutor, and that's how they really shine that turd.

28:27unknown Yeah.

28:28 And, you know, if if we could drop the scarcity and austerity mindset and think in terms of what would it be like if we wanted to sort of richly invest in education? There's so much that we could do. And I have I have a favorite passage from the book that I'm not gonna read the whole thing of, but this started as a newsletter post that I authored, just in frustration at all of these universities putting out their AI statements about how we're gonna be the leader in AI, we're gonna, you know, invest in it throughout.

28:54 And it's like, I said, what if, what if just one university would say instead, and this is this was a point of an edit in the book where I had, um, so the introduction is imagine if just one two or four-year college put out a statement along the following lines, and the original version of that had O-N-E and then parentheses, you know, digit numeral one.

29:11 And Alex is like, this joke isn't gonna fly. So that's out of the book. But the the imagined university statement ends with, we recommit to our mission of training students to be critical thinkers across disciplines, who can consider sources of information and locate them within their context, who can evaluate toolkits for the tasks they are taking on and decide which tools fit which task, and who can see through the glib marketing that power cloaks itself in.

29:34 And I would love to see that, or something along those lines, where you know, we really inhabit education for what it is. And that to me sort of speaks to this larger thing that we can do in resisting AI hype, which is to really make vivid what it is that we all do in our lives and in our work. For some people, art is work, um, but not everyone, right?

29:54 And when education is sort of held up as, oh, a chatbot can do it, it completely devalues what it is that we do as educators. And that same thing happens over and over and over again in all of the fields that are supposedly automatable via AI.

30:07 It's important to recognize or or to to aspire to want to promote that critical thinking ability in as many students as possible, but also to promote that notion that people should be able to assess the technologies in front of them or, you know, the different tools and systems in front of them to see what is working for them and what is not, to be able to properly do that, right?

30:25 Before we get to those kind of broader ways to address, you know, the AI hype and the harms that come with it, I wanted to ask you both a broader question. You know, I asked about education and the rollout of generative AI there. But what are your kind of biggest concerns that you've seen with the AI rollout over the past couple of years?

30:42 There's so much different things. I mean, and generally the kind of areas that I'm most worried about are the just the kind of question of how it is being used to displace work or, you know, and being promised to do a whole bunch of different things in different forms of labor. In the book, we talk a lot about journalism and science and medicine.

31:06 And I think the last one is probably the one I am the most worried about. I think although social services are kind of a close second, medicine worries me the most because I think that's where there's probably the most excitement and the most money. Medical AI startup is is a phrase that I've seen way too much.

31:29 But I mean, that has real immediate effects, right? And we've it's sad that at this point we have at more than one story where someone was speaking with a chat bot. It said one thing or whatever, led them to kind of a psychiatric crisis, and they they took their own life. And that's very upsetting. I mean, that happened with a character.ai bot.

31:56 It happened with uh this bot that was developed by Chai Research, and we mentioned this in the book, I think at the outset of chapter four. That's a pretty alarming thing. There's more of those forming, not fewer, right? And we're seeing these in not just in mental health, but also in skilled nursing and different kinds of clinical settings.

32:16 It's not a thing that professionals, nursing or medical professionals want. These are things that people in that National Nurses United have done so much political education about. They're like, we don't want these in our in our offices. My sister's a nursing practitioner, and she's like, I don't want this stuff here to be with me in the clinician's office, right?

32:38 I feel like the United States is probably going to be like an innovator with scare quotes around innovator on that because of the private healthcare system, right? I feel like even before the generative AI moment, you already saw Amazon and a number of these other tech companies trying to find their kind of niche in healthcare to tap into these profits.

32:56 And so unfortunately, it's no surprise to see them trying to use generative AI for that as well.

33:01 Yeah, well, just to add to that for a moment, I think that you're you're right that, you know, the disaster that is our healthcare system makes it ripe for this. But I also think that those with nationalized healthcare are not necessarily safe because it can look like cost cutting. And you you were just telling us before we started recording that you have a new minister for AI in your national government.

33:17 And so, like, you know, what's going to come? And I worry for the UK as well along those lines.

33:21 Oh, and and I would just say I completely agree with you. I don't think we're free from this at all.

33:26 Yeah, unfortunately. Um, but so so yeah, I mean, healthcare is one really concerning area. I have a hard time ranking these things. Um, another area that is super concerning because it is, I think, invisible to much of the public is the use of so-called AI systems in warfare. And I forget where I came across this line, but but somebody was saying, well, I mean, surely autonomous weapons no one no one wants war, but so long as we have war, then wouldn't autonomous weapons be good because they keep people out of harm's way?

33:57 I'm like, who is being kept out of harm's way, right? That was a potential question that someone was Oh, in some upcoming podcasts we haven't done that one yet, I think.

34:05 I think there was something about that in one of Andreessen's kind of manifestos as well.

34:09 That seems very likely, yeah. I mean, also very dehumanizing because the people that the autonomous weapons are attacking are not considered human.

34:18 Exactly. The people the weapons are pointed at are yeah. Uh so you know, there there's an awful lot of stuff there, and there's a a really big potential for sort of displacing accountability. And so there's some great reporting in 972 magazine talking about how Israel in their war on Gaza created this like target identification system, and the people behind it are like, this is great because it allows us to generate targets faster.

34:42 So like kill more people, and oops, if we get the wrong person, well, the computer told us to, kind of a situation. And I think a lot of that is, you know, less immediately visible and often bound up in, well, it's national security. We have to keep we have to do this to stay safe. And also bound up in the long history of AI, where you know the start of the project was very much military funding.

35:04 Yeah, no, it it's a very important piece of this, and one that I feel like doesn't get near the the attention that it deserves. And certainly, as you say, we've seen that great reporting on how AI is being used to justify increasing death in Gaza and I'm sure the West Bank as well. And I have no doubt we're seeing that in other areas of conflict around the world too, and we're just going to see that expand as these tools are continuing to be promoted in that way.

35:29 I want to pivot to what you discussed near the end of the book because this is something that I thought was really interesting and is probably something that we haven't discussed as much on the show. And this is how we think about addressing AI hype and these broader harms. And you kind of go through multiple layers of potential points to target these particular systems.

35:49 And so I wanted to start by talking about regular people, right? I think there's often a recognition, and I talk about this on the show a lot, that to really address these things, there need to be broader structural changes, right? You need policy to target these things. It's individual actions aren't always going to be able to take it on.

36:05 But I feel like on the flip side, that doesn't mean that there's nothing individuals can do to try to push back against this, to try to educate themselves, to try to think about what they are doing. So I wonder on that front, right, what do you what do you think that individuals can do when they're thinking about AI hype and the rollout of these tools?

36:20 One of the strategies is just around asking questions. If someone is encouraging you to use an AI system, encouraging you to buy an AI system, maybe you are actually in a decision-making thing, or maybe you are doing some community advocacy. Tamana Petty, for example, is an amazing leader who's managed to push back on various facial recognition technologies.

36:38 And we can all ask questions like this privately and in public. So when we are have some proposed automation or it's been in place and we want to fight back against it, the first question is, okay, what's being automated, right? What's the input? What's the output? And can you reasonably connect them to each other?

36:55 So if the input is audio and the output is text, and in the audio there was someone speaking, and the text is meant to be a transcription of what they said, that is a reasonable task. It might not work equally well, and there's lots of things to worry about when it goes wrong, but that's a reasonable thing.

37:12 But in contrast, you sometimes see these research projects where people claim that they can tell from a picture of someone's face if they're a criminal or not. There is no sensible connection between input and output there. And so that is a place where you can say this this does not make sense. You can also look at how the systems are being described, right?

37:28 If you've got this anthropomorphizing language, then that is a really clear sign that they are being sold as doing more than they can actually do. And that's sort of a place to push back. You can ask, how is it evaluated? Do we know how well this system works in some test conditions? And then how do those test conditions relate to how it's actually being used in this environment where I'm considering it or it's being used on me or whatever?

37:50 You can ask who's benefiting, who's harmed, and if something goes wrong, what's the recourse that's in place? And these are all questions that we can all ask and use them to make informed decisions, but also help other people, if we are not the decision makers, make informed decisions. And so that is sort of the serious work.

38:09 But the the main message that I want to get across is you don't have to be the kind of person who can build the system to ask these questions. And oftentimes the relevant expertise doesn't sit with the people who build the systems, but it sits with the people who are existing in the environment where the automation is happening.

38:22 But you can also have fun, right?

38:25 Right. And so, I mean, in that case, there's the different kinds of things for the individuals to do. There's the thing that we like to call ridiculous praxis, which is a thing that we that came out of the podcast, in which if there's something that makes no sense that is absolutely ridiculous on the face of it, you should make fun of it.

38:45 And I know you do this quite well online, Paris. Um, and I'm sure listeners of your show do as well. But I mean, if it just makes absolutely no sense, make fun of it. If somebody posts a silly, ridiculous, cheap looking AI generated photo, you know, troll the hell out of them. You know, I would even do it for things that ostensibly look good, like this studio Ghibli, Ghibli.

39:10 I said it in a way on another podcast, and Emily trolled me. Like it was glee, no, ghibli, studio Ghibli. So yeah, Studio Ghibli. I mean, make fun of them say, hey, do you know how much you know energy it took to generate that? And do you know that this many people did that? You know how much water that probably took?

39:27 You know how much, I mean, I it's worth really digging in and also because it is not good quality. I mean, it's what Tressie McMillan Cotton had um the uh New York Times. I know she had a column on it, but she puts it on on Blue Sky, like it's mid, it's bad looking, you know. I mean, it's effective. I mean, there's a cultural weight to making it uncool, to making it seem out of fashion.

39:51 There's also strategic refusal. I mean, just not using it. I mean, if it's I know there's pressures to use it from all corners, whether you're a student or a worker or whatever, but if you make a principled stand, I mean, it does have an effect and it can inspire and and and I think relate to other folks to do it.

40:10 And then I think there's those collective solutions. I mean, when we talk about policy in the book, and we're in a you know, terrible policy environment in the US right now, and you know, the states are a little better, but you know, we have our own battles in California and and and elsewhere, including this thing that might sneak itself into, I don't know when you're going to publish this podcast, but there's a thing in the that's under consideration in the in the US House right now where Republicans want to prohibit any kind of AI regulation for 10 years to be enacted by states.

40:43 Um and they're trying to sneak that in through the budget reconciliation process, which means they don't have to, they don't have to get the the necessary votes that would, they don't have to invoke cloture and it can't be, it would have to be, they'd only have to get 51 votes. And so, you know, there's those policy solutions, but the collective ones I think are very powerful.

41:02 So, what are collective ways in which we can push back, whether that's through our unions, through our communities, through our, through our schools and our clubs and whatnot, what are ways that we can really lean into relationships and really make a stand and say we're not going to rely on these technologies in our cultural production, in our operations work, and really pushing back can be very, very effective there.

41:26 And so I think the same way that people are relying on each other for things like mutual aid and kind of mutual support. I mean, we can think about ways of doing that as kind of consumers and workers.

41:35 Yeah. And I feel like given the state of things in the United States right now, that is probably a really good path to try to go down, especially seeing politically and legislatively what might be occurring. The ability to kind of get people together to use unions and other organizing structures to push back against this seems really opportune.

41:53 I just wanted to add that on the sort of individual but also inspiring people around us point, that there's this inevitability narrative that we hear from the tech companies all the time. And that is a bid to steal our agency. And so, you know, Alex said it can be inspiring to take a principled stand.

42:08 And one part of that is to say, no, I don't believe you. If you say, well, this is here, we have to learn to live with it. No, right? There's nothing inevitable either in the development of something called AI or AGI or in the sort of continued presence of this tech in our lives. And to sort of claim that space is really important, especially as you were both saying, in this context where our policy environment is so terrible.

42:29 Yeah, really important point. And I was just going to pick up on a couple of the other things that you were saying. Like in terms of being able to ask the questions, that's where that kind of ability to have that critical thinking comes from, right? You need to know what to be able to ask and you need to be kind of trained to get an idea of how to interrogate these sorts of things.

42:46 And then, of course, on the strategic refusal, people are sometimes surprised when they hear I've never used a chatbot, or at least, you know, willingly, sometimes I've had to engage with these customer service chatbots because that was the only option that I had, at least to try to get to someone human, right?

43:00 But I read how other people assess them, and I don't feel that I need to go and do my own little personal assessment too. Uber is another company that that I've never used, despite how common that is in society. So people can do these things, and I don't think I'm changing the world by doing that, but I just feel that that is a decision I have personally made.

43:19 I'm totally with you on not using chatbots. I do my very best not to consume synthetic media. I sometimes get trapped. Like people will will send me something and I don't realize quickly enough. And like, oh, I've heard some of that, I don't want to listen. Or when Notebook LM came out from Google, there was a couple of journalists who wanted me to comment on one each of those fake podcasts, and so I had to listen to them.

43:39 But for the most part, I will not willingly consume synthetic media. And people are surprised. And they like to say, Well, how can you possibly judge? I'm like, Well, I can tell you a lot about how these things work and about how we work with them. I don't have to subject myself to it to make those arguments.

43:55 Yeah. And I mean, I wrote a newsletter post that was that was something like, I hate, I hate how. Having to look and trying to discern whether something is AI slop or not. Like it is not a useful skill insofar as it's useful insofar as like this is something that I cannot trust, but I'm so upset that there's so much of this here on the web.

44:16 Now I have to use cognitive energy to discern whether this is an AI generated image. And I have to then change my opinion of the brand because they don't respect me enough to actually use or put some put some effort into this or to hire somebody to do this. You know, same thing with text. I mean, you can discern it kind of reading long enough, but it's and I think this happens a lot when we um we get lots of podcast pitches, as I'm sure you do too.

44:43 And I'm like, uh, this this reminds me of synthetic text. This seems this seems like somebody did not do their work here.

44:50 Yeah, I'm sure so many of those pitches are are generated and just sent out en masse. I have no doubt about it. We need to boycott Coca-Cola now, right? For uh using generative AI in its ads and forcing that on so many of us. I I wanted to move to talk about legislation as well, right? Because you you talk in the book about how there are existing laws on the books that we can use, but there's also new legislation that can really help to challenge AI hype in important ways.

45:16 So I was hoping you could talk a bit about the legislative front and how you think about that in this moment, given what is happening in the United States.

45:23 Just to back up a little bit, there's a few tropes that come from the tech lobbyists to the general public and to policymakers that I think it's really important to push back on. The first one is tech is moving so fast policymakers can't possibly keep up. And related to that, this is something that only the people who build the tech understand.

45:41 That's absolutely false, right? The point of policy is to protect rights, and that's not moving so fast. I mean, in some cases, we would like it to move a little bit faster. But the question is not, you know, how do we regulate this technology, but how do we protect rights given changes in the environment that are due to the technology?

45:57 And that starts with how do we use the policy that we already have. Um and under Lena Khan, the US FTC was doing some fantastic work, sort of predicated on the idea that their role is to regulate business activities. And if those businesses are using automation to do those activities, that doesn't change anything.

46:13 So there's there's a lot that can be done with existing regulation. Another one of the tropes is that this is a race. Usually within the US, it's you know framed as a race between the US and China, you know, very cyanophobic framing. And if we, you know, hamper ourselves with regulation, then we're not gonna win the race.

46:33 So you know, that is one to push back on. And then finally there's this trope that uh regulation stifles innovation, which is connected to the the second point there. And that's absolutely false, right? Regulation harnesses, shapes, guides innovation towards the goals of the regulation. And the only thing that it stifles is the ability to just like amass all the money, which uh kind of is the point, right?

46:55 And and you can't use innovation to refer to that and be honest at the same time. So I think there is scope for protecting rights more thoroughly. I think the status of privacy law in the US is is really not where it needs to be. And, you know, there's principles of like data minimization that I would love to see more strong.

47:16 And then my sort of like pipe dream uh regulation would be to set it up so that anybody who produces a chatbot is actually accountable for the output of that chatbot. Because a lot of our issues with this come from the fact that nobody has accountability for what's coming out of the chatbot in most cases.

47:32 There was the um Air Canada situation where Air Caraca actually got held accountable for their chatbot. Um, but for the most part, that's not well established.

47:40 I don't know if you saw, but there was a recent story, I think, in the Financial Times, that insurers are now offering uh new type of insurance to companies when their chatbots make errors or mistakes or say things that end up costing them a lot of money. Right.

47:53 Yeah, which that's interesting. I guess I like that it suggests that the company should be responsible for it. But I think there's also interesting questions about like how do we allocate accountability? So if you think about the instances of the fake mushroom hunting books that get posted on Amazon and people actually getting injured because they've followed that, it's like, okay, so how do you allocate the accountability among the chatbot producer?

48:16 So, you know, whichever system was used to produce the thing in the first place, to the person who used that to produce the book and put it on Amazon to Amazon. Like, you know, sort of where is the accountability across those those three sites? I think that's an interesting place to be working in.

48:29 Yeah. The state of policy in the US is pretty is pretty rough. I mean, there's on the federal level, it's pretty much stopped, right? On the state level, there are a few proposals. So, I mean, there have been some things in the last California legislative session. There were some things that were targeting things like non-conceptual deepfakes.

48:49 There was something that was that SAG Afra was promoting in terms of preventing likenesses from being used that didn't prevent them outright, but did put some guardrails around that. Right now, Assembly member Bauer Kahan has a bill called AB412, which would at least allow copyright holders to make a request of AI model developers to get notice of if any of their copyrighted works were in the training data, and they actually have to respond within seven days.

49:22 If they don't respond, then they can file a civil suit. So there's something. I mean, it's it's it is something on the copyright front, which is again a pretty weak mechanism, but it's kind of the only one in which artists have been able to file any kind of grievance with companies, and you're probably quite aware of the suits that many people have taken, including the New York Times and the large lawsuit by Sarah Silverman and Jody Picot and Game of Thrones guy, whose name I'm forgetting right now.

49:51 George R. R. Martin, sorry. But they're at the same time, you have this attack on the legislative front that I mentioned earlier that the Republicans are are trying to force through the budget bill. There's also attacks on the California Privacy Commission right now, which they've got rid of the person who was the executive director and have installed someone who's much more middle of the road, much more industry friendly.

50:13 And so you've got, you know, these things happening. And so anything that's targeting AI, particularly, has been sort of defanged. But as Emily said, there are existing, existing laws on the books, and if it's going to be the kind of matter of if there are people in regulatory agencies who are willing to enforce them.

50:34 Yeah, you know, obviously I agreed with all of the proposals and arguments that you made around legislation. The one that really stood out to me was thinking about not just data rights, but minimization of data collection, right? Because we know that these systems rely on all of this data that has been created and that they have kind of scraped off of the internet to create these models.

50:53 And I think that's a real point to think about pushing back on is just how much data these companies can collect, can create, can store on us, and trying to push back against the power that comes with that is one thing that I think would be a great thing to look more at into the future. But we're running close to the end of our time.

51:08 So I just wanted to kind of end off by asking you a broader point. You've you've put the work into this book, you've been hosting your podcast and being really vocal critics on AI for the past few years as we have entered this new hype cycle around generative AI. And so I was wondering, you know, what you most want people to take away from reading this book uh, you know, once they pick it up.

51:30 That's a great question. And I and I want to have sort of a pithy answer to it, but but I I think that I think that I want people to take away that we all have agency here. And that it doesn't matter if you are not a tech developer. It doesn't matter if you feel like you're bad at math and computers, your expertise is still relevant.

51:47 You still have something to say, and you still have, you know, ground to stand on for your own refusal and your own decisions.

51:54 I mean, my PV thing is that AI is not inevitable. It's a sort of tagline that we also have at Dare. And I think is the one that this, you know, this book re-inscribes and and and provides a little bit more meats on how to push back. And so, you know, I want folks to go away and know that amongst the horrors of the horrors of the world, AI surely is not inevitable.

52:15 And part of one's own kind of political action, you can incorporate this pushback as well.

52:22 It's been great getting to know both of you the past few years. I'm really excited for people to pick up the book, to read it, to engage with it. I know it's going to be helpful for so many people and getting a good perspective of the critical view on AI. Thank you both so much for taking the time to come on the show.

52:36 I really appreciate it.

52:37 Thank you, Paris. Thanks, Paris. It's so great to talk with you again.

52:43 Alex Hannah and Emily M. Bender are the authors of the AI Con. Tech Won't Save Us is made in partnership with The Nation magazine and is hosted by me, Paris Marks. Production is by Kyla Hewson. Tech Won's Save Us relies on the support of listeners like you to keep providing critical perspectives on the tech industry.

52:57 You can join hundreds of other supporters by going to patreon.com slash tech won't save us and making a pledge of your own. Thanks for listening and make sure to come back next week.