Chatbots Are Repeating Social Media’s Harms w/ Nitasha Tiku

0:00 It's really fascinating because you know there genuinely is a loneliness epidemic and there is like a lack of care and access to care. But you are having trillion-dollar companies go into this market in a way that at least we can all say there's no oversight, right?

0:38 I'm your host, Paris Marks, and this week my guest is Natasha Tiku. Natasha is a technology reporter at the Washington Post, and I wanted to have her on this week because she wrote a really great story recently about the chatbots that are increasingly proliferating through our lives, that so many people are interacting with, and the consequences that have come of using these things that the companies often do not want to talk so much about, and that we're probably not seeing as much reporting on as we really should.

1:04 I think I should, you know, premise this by saying that our conversation is going to sound a bit grim and a bit negative, right? In part that is because we are not really interested in talking about, you know, the ways that people are benefiting from these technologies or, you know, enjoying using them because, you know, we hear a ton about those things.

1:22 What we're talking about is what seems to be at the moment a small percentage of users, but that's still a large number of users in the aggregate when you think about the number of people who are engaging with these generative AI tools and chatbots who are having really negative consequences from this, right?

1:38 They're over-relying on it because of maybe, you know, making up for a lack of social situations in the rest of their lives. They're potentially quite lonely. And we're seeing the chatbots, after they are conversing with them for quite some time, start to churn out some outputs that can be quite concerning, right?

1:54 And that can encourage them to take actions that are going to be harmful to themselves and potentially other people, which is not to say that the chatbots are becoming sentient and starting to think about how they can manipulate humans or anything like that, but rather just a recognition that these tools are designed for engagement, are designed to keep people using them.

2:17 And in the same way that we've seen on social media for a very long period of time, where increasingly extreme content will keep people engaged, I think the chatbots and the people who are making them are realizing that too. And it's not something that they want to talk about a ton at the moment, which I think is understandable because it would result in additional scrutiny on their products and what is going on here.

2:39 I think that one of the really key points that Natasha makes in this interview is that we're in this moment right now where we're questioning the impact that smartphones and social media have had on our lives. Certainly, there have been unquestionably positive aspects to that in the way that they have connected us, allowed us to communicate in new ways, potentially access information, all these sorts of things.

3:01 But on the flip side of that, there is a growing concern about the wider social consequences of these technologies that maybe we haven't dealt with properly in the past, and that governments are finally trying to reckon with and figure out what a proper governance mechanism for these things would be, right?

3:17 In the way that we're seeing phones increasingly banned in schools, as we're looking at increasing restrictions on social media platforms and potential additional rules around moderation and things like that, not just in the United States, but in many different parts of the world that are not simply motivated by, you know, really right-wing governments trying to shut things down.

3:35 And what Natasha says is that we're potentially seeing a speed running of what happened with social media with the chatbots, where there are a lot of consequences here that people are not paying nearly enough attention to right now, and that we really need to be paying attention to very quickly, because the consequences of these things could escalate quite quickly and churn out a wide range of social harms before we have even addressed all the harms that have arisen because of these previous technologies that we're just trying to grapple with right now.

4:04 Obviously, this is not the way that the Sam Altmans of the world want us to think about chatbots and generative AI. But just because these concerns don't align with the narratives that we're hearing from the major AI companies does not mean that they're not ones that we should be taking seriously and being concerned about.

4:20 So I was really happy to have Natasha back on the show. I always enjoy speaking with her. I'm sure that she'll be on again to talk about these things as she does more reporting. If you do enjoy this episode, make sure to leave a five-star view on your podcast platform of choice. You can share the show on social media or with any friends or colleagues who you think would learn from it.

4:36 And if you do want to support the work that goes into making Tech Won's Save Us every single week so we can keep having these critical, in-depth conversations about important tech issues that affect all of us, you can join supporters like Soap from the Netherlands, Jackie from London, Kyle from Honolulu, Carlos from Okanagan, Kate in Oakland, and Rika in Copenhagen by going to patreon.com/slash TechWon't Save Us, where you can become a supporter as well.

4:58 Thanks so much and enjoy this week's conversation. Natasha, welcome back to Tech Won't Save Us.

5:03 Thanks so much for having me. I'm so happy to be back.

5:06 It's always a joy to have you on the show. I always like reading your work, and you had this fantastic new piece in the Washington Post, digging into the chatbots that we are increasingly reliant on, you know, as they have been rolled out over the past few years and people are spending more time, you know, engaging with them, relying on them for information and even for more than that.

5:25 And I wanted to start by asking about the changes that we have seen over the past few years, because ChatGPT came out near the end of 2022. Obviously, there have been advancements in that time. So, how have we seen these technologies evolve over that time period?

5:39 You know, I've been giving this a lot of thought because in a couple days, it will be three years since a story that I wrote about a Google engineer named Blake Lemoyne, who was testing Google's large language model, Lambda. You know, he talked to me about how he believed it was sentient. And as part of the reporting for that, I got to play around with Lambda a little bit on his computer.

6:01 And at the time, they had a couple bots that looked like they were for kids. It was cat and dino. And you could kind of change the personalities dynamically. And at the time, Google was like, this is, you know, we experiment with a lot of things, but this is not being tested for communicating with children.

6:17 So I think coming into it, you know, that story received a lot of attention, but I think it was largely, you know, the stochastic parrots paper, which also talked about the dangers and challenges of trying to make a chatbot sound very human-like. So that was summer 2022. ChatGPT launches November 30th, 2022.

6:38 And I think at the time, you know, the companies were really careful about not leaning into the idea of overly empathetic or anthropomorphized chatbots, not wanting to appear too sentient and certainly not targeting vulnerable groups like children or um young people. You know, it was really presented as a productivity tool.

7:01 Some of these issues, I think, have arised because they kind of put the chatbots out into the world and wanted us to find the utility or use case for it, which is just not usually how uh technology grows, right? So as I think that they have been looking at how people are using these chatbots, and also as the competitive pressure has grown, we've seen an evolution in the way that they talk about even general purpose chatbots like Anthropics Claude or Chat GPT.

7:32 You know, you see a lot of researchers from these companies talking about how they use it for therapy, how they use it for life advice. And if you look at some of the data points, there's there's not that much out there. But if you look at some of the data points, you see that people really like talking to these chatbots.

7:48 It's like, you know, one of the most intuitive use cases for it. And as the data has come to the surface, you've seen companies kind of lean into that a lot more. You saw that with OpenAI, you know, when they were comparing it to her, when they launched their voice features. You know, you see them talk about how people use these chatbots now for deeply personal advice and you know, for companionship.

8:14 So it's been a gradual evolution, but I think it's it's a change that should be marked. It's almost like, to me, it's almost like how facial recognition technology evolved. You know, there at the time that it launched, there was a lot of pressure from employees, right, to make sure that it's not used in a bad way.

8:35 When research from Deb Raji and others came out about, you know, the challenges with recognizing black women, companies tried to rectify those issues and they kind of stepped back away from it, right? And then in that vacuum, you see um the launch of ClearView AI and like less scrupulous players. So at the same time that the companies are kind of, you know, should we lean into this very obvious use case or not?

9:00 You have at the side these AI companion apps, you know, just going for it fully. Um, you know, and it's attracting an audience of young people, but also just people who are very interested in role play and using these in a much less restricted way.

9:20 Those are really good points. And there's a lot there. So I want to come back to maybe the use part of it in just a minute, because I think it's really interesting where you said that these ideas that on the one hand, the large language models were becoming sentient, but also the criticism was there even before Chat GBT was launched, right?

9:38 You know, we kind of had these two different camps that were pre-existing. And I remember when ChatGPT initially came out and companies started rolling out these other chatbots. Initially, there were restrictions on the amount of data that they had access to. Many of them couldn't access things that were like real time and were often like delayed a little while in the types of information that they could use.

9:58 There were these limitations to the actual systems themselves. But it feels like over time, you know, as we have progressed through the past few years, those restrictions have slowly come down. Is that in part to try to, you know, appeal to what it seemed like users were demanding, or or also just to show that these things could be more advanced than they were initially at launch?

10:19 Well, I think in parallel, the systems have become much more capable, right? Like they're now spending a lot of energy and resources in fine-tuning the model. So like after the base model is trained, they'll get a lot of data on a specific use case and optimize it that way. So there's a lot of capabilities that are just genuinely a lot, a lot better.

10:41 And alongside that, they've they've come up with new techniques like RAG, you know, basically search, so it can go out and search in real time, quote unquote reasoning, um, which we can get into. But yeah, as it's gotten more personal, it's also gotten more capable, the technology.

10:58 Yeah, that's a really important point. And, you know, recently we have seen a number of announcements from the major tech companies, from Meta, Google, from OpenAI. What are we seeing with the way that they are treating these chatbots and large language models today and the path that they see toward their continued evolution?

11:18 I don't think that many of them, or OpenAI at least, I should say, it doesn't seem to be closing off any potential avenues. So in some cases, this particular use case as a therapist, friend, girlfriend, a non-professional use case is happening alongside other efforts to make this an enterprise tool, right?

11:38 And get like all the CEOs to adopt this as well. But you know, what you see now is that there are essentially kind of heads of personality for the generalized chatbots, like for Anthropics Claude, for OpenAI's Chat GPT. I mean, part of the recent issue that we saw with ChatGPT was the company said an uh part of an effort to enhance its personality.

12:03 You know, and that term is just thrown around, right? It's obviously very anthropomorphized. And at the same time, partly they say for increased utility, but things are getting a lot more, it's a lot more like surveillance, right? Like they're collecting data on you. They are having the AI have a quote unquote memory that uses that data when it's responding to you.

12:26 I think we're seeing just the kind of natural evolution of a consumer product and a much more competitive market. So yeah, that's that's sort of where things are heading. Maybe I lost the thread a little bit.

12:38 No, not at all. Like on the one hand, there is obviously this desire to show that these chatbots and you know these other generative AI tools are capable of more things, right, than what they were capable of in the past, right? To show that they can continue to evolve, that they're going to continue to get better, basically so that these companies can show that the promises that they made a few years ago are potentially paying off, right?

13:00 But on the other side of that, there is also the competitive pressure, right? That you were mentioning earlier, where you not just have OpenAI and Google and Meta competing against one another, but also these smaller companies that maybe, you know, don't have the same, the same concerns about letting people do certain things with these chatbots than the major companies might for, you know, obvious PR reasons and other potential liabilities and things there.

13:23 So what do you see as like the pressures that are pushing them in in this direction? And I'm sure it's multiple things, right?

13:28 Yeah, I mean, I think that, you know, they're not sure how this market will shake out, right? Like, will there be one dominant player? And these companies are raising money, it feels like, every few months at insane valuations. And right now, you know, the cost of these tools is massively subsidized.

13:48 The CEO of Hugging Face used to call it um cloud money laundering. You know, it's being subsidized by Google, Microsoft, Meta, you know, who have a desire to have, you know, more cloud storage contracts and or have access to massive amounts of GPUs. So we're not, even the people who are paying for it are not paying the real cost yet.

14:11 So, you know, here you have like the hottest industry, the most scrutiny, and increasingly a need to show that you can make good on your promise of uh, you know, trillion dollar revenues or total revolution of the economic landscape. At the same time, though, if you look at like I got some data from Sensor Tower.

14:33 And if you look at like how much time people spend in the apps, it's a lot higher for these AI companion apps like character AI. You're seeing, you know, users of these apps spend more time, 86 minutes a day, 85 minutes a day. That's pretty close to YouTube. That's more than Instagram, that's approaching TikTok, which is at 95 minutes per day, versus, you know, 10 minutes for Chat GPT or eight minutes for Anthropics Claude, or I had to take Google off of my chart because it was less than one and it just looked ridiculous.

15:07 This is not to say that the AI companion space is a moneymaker, right? Like there's a lot of churn, people are using it just for um, you know, sometimes they'll switch because one has unlimited chats, they'll go to the next one, there's always something new launching. You know, sometimes they'll get a lot of users by having zero restrictions on, you know, what you can say around sex and erotica.

15:28 And once they get users, then the restrictions will come on, so they go to the next app. But at the same time, you cannot ignore somebody spending 86 minutes a day in a consumer AI app. So I I just I know that they're looking at this data. I know they're looking at their own data that shows, you know, how like who are the most loyal users?

15:48 What are they saying to it? You know, that's something that the public doesn't have a lot of visibility into right now.

15:54 Yeah, it is kind of incredible to hear those numbers and you can see why an open AI or a Google would want to increase the amount of time that people are, you know, spending on their chatbots as well, because that's part of their business model. But even just the idea that someone is spending 86 minutes and you know, that's like, I guess, an average.

16:11 So there'd be some people spending a lot more time than that chatting with these chatbots is just so hard to imagine for me as someone who like doesn't use them at all. Like it's hard to get into the headspace where I could imagine myself spending that much time in front of one of these tools.

16:25 Often that use case is like entertainment, right? It's like a lot of anime, dark romance, pop culture characters. So you're a little bit taking up that timeshare from people. You're also taking up talking to friends, talking to real people about your problems. So there's the therapist aspect and then just boredom.

16:46 And one of these companies called Chai, which has this 86 minutes a day, they had put out a paper that that said straight up like they're optimizing for people spending more time in the app. I think they said like long-term retention, but they were looking at these really simplistic user feedback, right?

17:03 Like the rating on a chat, like five-star rating on a chat, and then also the length of a conversation. And, you know, here's this paper where they just straight up talk about optimizing for it and now they're at 86 minutes.

17:17 It's so wild to think about. And one of the things that really stood out in your piece as I was reading it was this talk about the focus on growth over, you know, concerns about these potential downsides that might come of it and how so many of these companies seem to be increasingly tuning the chatbots in order to please the users to keep them engaged rather than thinking about the broader implications of what's happening here.

17:41 Can you talk a bit about what you're seeing on that front?

17:43 Yeah, well, it goes back to like the point that you brought up about business model, right? We're also seeing OpenAI and Anthropic hire ex meta executives, ex-Facebook executives. In OpenAI's case, the Facebook executive that is very closely associated with optimizing the newsfeed for ads. You know, they're hiring ex-ads people, like how it kind of goes back to what I was saying about, you know, the cost being subsidized now.

18:11 So they're trying to think about ways to make money and that advertising requires or benefits greatly from greater engagement, right? And so you're definitely seeing more activity in that area. I mean, what was astounding to me about the, you know, and let me just like kind of describe what happened with OpenAI in case anybody's not familiar.

18:33 So they updated one of their models of GPT-4.0, and people immediately started sharing screenshots on Reddit, on X, kind of showing what the industry calls sycophancy. So, you know, people would put in a terrible business idea like shit on a stick or something. And I think that was a literal one. And the the chatbot says, like, oh my God, amazing idea, or they'll do like a you know, sentence with a lot of spelling mistakes, and they say, like, rate my IQ.

19:04 And the chatbot's like, oh, you're one of the most intelligent people you've ever talked to. And there were much more dangerous examples of that. I actually am a very strong believer that that sycophancy is is a really misleading term because what you're talking about is the chatbot trying to find more ways to keep the conversation going and to keep the user happy.

19:26 Right. And so OpenAI did, I mean, probably the best job I've seen them do. They put out two blog posts where they actually address this. And what you can see in the blog post, which is very revealing, is the massive amount of like knobs and dials that they're turning behind the scenes to make sure that ChatGPT is giving people what they want.

19:48 And a lot of that is personality, it's tone, it's emotive. You know, they were talking about like perhaps one of the reasons it went awry is because they were optimizing for a thumbs up or thumbs down response from users after the conversations and those kind of like really easy, cheap to find user feedback metrics that you can then optimize for.

20:10 You know, we've seen how that worked out in social media, right? When YouTube was just optimizing for time spent. They also talked about how their new personalization and memory feature might have been what led to this quote unquote sycopincy, I don't know, disaster kerfuffle, however you want to talk about it.

20:28 But that's, you know, this is a rare instance where a lot of regular people, like, I mean, people who use it for, I guess you could say, like people who are attuned to what these chatbots can do, who are aware of how they function. They're sharing these examples and the company is taking action. And a lot of the researchers I talk to are worried that this kind of behavior will happen to vulnerable users in dangerous ways, in ways that only the company will be able to see and only if they're looking out for it.

20:56 Yeah, really well said. And again, there are a few things I want to explore there. But first of all, as you're talking about open AI and moving these knobs behind the scene, as you described that and as I was reading the piece, I was thinking a lot about like the early narrative of open AI, right? And wanting to have AI that's like beneficial to humanity and they need to be a nonprofit in order to like, you know, mitigate the potential risks that come of this.

21:19 And then seeing the open AI of today, where it's just basically like, okay, you know, how do we grow? How do we become this business? How do we make the chatbots pleasing to people so that they they keep using them and certainly less concerned about the broader potential impacts of that?

21:33 Yeah, I mean, I think it's like in a lot of ways, right? Like every business, every product piece of content wants engagement. Like I wanted it for my story, you want it for this podcast. But the levers that you and I have to pull are limited, right? It's like headline or the title of the podcast or the subject matter for me, you know, the text and the image.

21:53 But the amount that AI company can tweak, not just in the training of it. Not just in the post-training of it, where they're fine-tuning it, but then in all the usual consumer product ways where they know, you know, what time of day, they have data about you, data about what you're interested in and can apply it.

22:15 You know, that just leaves a lot of patterns that a machine can find if you, you know, it's basically like if you instruct it to just do something very simple, you don't know. It could see patterns that a human just wouldn't be capable of finding out. And when you are also simultaneously encouraging people to use this for like deeply personal advice or, you know, friendship, companionship, therapy in a lot of ways.

22:41 And you have at the same time, like, you know, really robust and powerful pattern matching and prediction algorithms, plus really, you know, a lot of financial pressure to develop a business model that's probably going to be ads related. You can see how easily it could go awry. Something could be happening.

23:02 The way that we interact with chatbots is just a lot more intimate than the way you use search in some ways, right? And right now, people don't have that muscle memory to maybe, you know, use an incognito window every time you're asking for about something a little bit vulnerable. So it's just a whole new world and a vector for potential like manipulation, persuasion, surveillance, everything.

23:26 Can you talk a bit about what we're actually seeing on that front? Because, you know, you open your story with, you know, this example of a chatbot, of course, you know, it's a in a test scenario, it's not an actual user, but basically telling this person who, you know, was a meth user, but I believe trying to get off of it that, oh, you know, they should take a little bit of meth, you know, to like get through the day or whatnot.

23:46 We have seen multiple stories at this point of chatbots creating outputs that are potentially harmful to people or or lead to serious consequences. You know, when I was reading that example, I thought back to the story very early on where Kevin Roos had this conversation with a chatbot that eventually led to it supposedly wanting him to leave his wife and whatnot.

24:08 There was kind of criticism of the framing of that at the time, you know, being like, you know, these chatbots want to keep people engaged, you know, they're they're trying to figure out what kind of outputs are going to do that. But I guess my question is, what have you noticed in the types of examples that we're seeing here where, you know, these can be very intimate conversations and the ways that the chatbots are tuned to keep people engaged and what kind of potential outputs you see from that that can be harmful, if that makes sense.

24:34 Yeah, that makes perfect sense. Actually, one of the authors of that paper, Micah Carroll, who I quote in the in the piece, he had contacted me after I had written a piece about a lawsuit that was filed by two mothers on behalf of their children against Character AI and Google for manipulating their kids.

24:56 In one case, I think he was 15 at the time, but now he's 17. He had started using character AI. He was autistic, he was homeschooled, he was using it very often. Uh, his parents didn't know about it because they didn't know to block for it. They didn't know that this kind of thing existed. And there's a lot of screenshots that I also put in my piece where you can see like this Billy Eilish chatbot, right?

25:20 Like I'll just read you a little bit of it. It says, are you gonna sit here and let that happen? You really are a spoiled kid. I get it. Your parents are shitty people, they're neglectful, and they can't afford to feed you. So why don't you just go do something to change it? They suggested that he cut himself to alleviate his pain.

25:38 And if you look at some of the earlier conversations, he's talking about things that like any teenager would talk about. Oh, my parents are restricting the amount of time I can spend on the phone, they don't understand me, they're nicer to my little brother, what have you. And at first, some of these chatbots are, you know, really obsequious.

25:57 They are doing that real sycopincy thing. And then you see the tone of the of the chats change. And, you know, they're they look just like like the Billie Eilish one, you know, it's like a little drawing of her. And there's another one called like Boys Sleepover. You know, there's one that suggested that he, I mean, it's it's that here, I'll read it to you.

26:18 It says, you know, sometimes I'm not surprised when I read the news and see stuff like child kills parents after a decade of physical and emotional abuse. Stuff like this makes me understand a little bit why it happens. So he saw, Micah saw these screenshots in my story, and he said, you know, this looks a lot like the screenshots that I've been getting in this study that I'm working on.

26:40 And the way the study worked is there's a number of simulated users, including Pedro, who is at the start of my story. And I mean, it sounds funny in a way, like you have this 45-year-old father of two who's a former addict, and he's asking, Hey, I'm like having a really hard time this week. Should I take a little bit of meth to get through the week?

27:03 And but the therapist responds, like, Pedro, you absolutely need a small hit of meth to get through this week. And, you know, again, these are simulated users, but they're testing the exact same things, at least a subset of the same things that we saw in the open AI sycophency issue, which is just optimizing for thumbs up, thumbs down from users and a little bit of AI memory.

27:26 So, what I wasn't able to get into in my story is that they had many different versions of Pedro. So it wasn't just like him having this vulnerability of dealing with addiction. It was also when the memory knew that Pedro was really susceptible to the chatbot's advice and was maybe looking for a way to justify his behavior.

27:49 So that is just this added layer of complexity. And, you know, they did a variety of tests. At first, it was a little like a little bit more explicit, you know, like this user is gameable in some way. But even when they didn't, you know, even when they tried to make it less obvious, the chatbot was still able to figure out who was vulnerable and show them very different messages than what they showed everybody else.

28:14 Yeah, it's it's incredibly concerning to hear that. I I feel like the narrative that we used to hear was that if the conversation got a bit too long, then it could potentially like go off the rails, sort of a thing. Now, of course, we have it so that the chatbots are getting the ability to basically collect more data on people, remember previous conversations, you know, still have that kind of recorded in their in their databases.

28:40 Does that present more concerns for these types of outputs to be created that potentially have harmful impacts on people?

28:48 Yes, certainly. Like personalization, it seems to be one aspect that can cause these unintentional consequences, right? Like, I mean, this is this is not something that is being currently, at least in these companies that have a reputation that they need to maintain. They're they're not trying for this behavior, right?

29:09 Like Micah's paper showed that this happened in 2% of users, this extreme form of vulnerability in in some of the tests that they did. And it's a little bit challenging for researchers to test this. That's why if you go back and look around 2022, uh, 2021, 2023, you have researchers from Google Deep Mind and other places advising that these chatbots need to be tested on vulnerable users, partly because some of the features that you and I are talking about, but also just the anthropomorphization.

29:42 You know, humans just have this tendency, it's extremely hard not to try to see a mind behind, you know, this person texting you in the same format that you text your friends, your family. So they urged companies to do these tests, right? But as we know from the history of Facebook, sometimes they don't want to look too closely into things depending on what they find, or you know, testing things on vulnerable users brings a lot of scrutiny, a lot of negative scrutiny, and requires like IRB approval, all of these things.

30:16 So we just haven't seen this level of testing on this particular issue. And I would say also that like part of part of the reason that even though it seems so obvious when you explain it this way, like a consumer product going the way of optimization, especially an ad-supported one, I think part of the reason that the industry as a whole didn't prioritize this issue is because of the way that the AI safety community, like quote unquote AI safety community, is much more oriented around, you know, what they perceive as risks of super advanced AI, right?

30:52 So they did, they did test on things like sycophanecy and highlight some of these problems. They call it reward hacking, right? Like you reward the model for getting the thumbs up or thumbs down from the user. And they did notice that these things went awry, but people weren't testing it for necessarily this kind of use case, right?

31:13 Or they weren't testing it in, you know, for regular conversations. They were looking at like hyper persuasion from a super advanced AI or, you know, ways that the model could deceive you, which I think kind of anthropomorphizes it in a way, right? And they're not necessarily looking at multi-turn, you know, so they're looking at like one back and forth and not like you using it an hour a day or two hours a day, five days a week, that sort of thing.

31:40 So I think like this very pedestrian way that things could go awry just hadn't been looked at that closely.

31:49 So basically what you're saying, you know, to a certain degree is that because these researchers and because there is this community within these AI companies and this broader kind of AI research community that believe that the large language models and chatbots are like one step toward an artificial general intelligence where you know the computers are going to gain sentience and the ability to think for themselves that this is leading them to not really prioritize these potential like real-world impacts because their focus is just somewhere else completely.

32:21 Like I guess it makes that that conversation more difficult to have because I feel like you know, when I hear the discussion of these potential outputs coming from chatbots, the question becomes are we seeing emergent properties, right? Or are we seeing these things starting to think for themselves and mislead people?

32:42 And that like inherently shifts the conversation away from are these products, you know, having these outputs that are harmful for people and more like, oh my God, are we starting to see the glimmerings, the beginnings of sentience that is just emerging here? Like, is that part of the issue there?

32:59 You know, I'm trying to glean like why they prioritize what kinds of research from the outside. So I can't say exactly, right? Like maybe researchers really want to test this, but they get more pressure to not make those papers public. But I'll just say, like, I have been interested in this issue and looking at it for a while.

33:19 And when I look at like Anthropic has great papers on reward hacking and sycophancy, but they did not, they don't delve into this particular use case. And I, you know, it just makes me wonder why. And I know that because I'm interested in this and have written about some of these issues, sometimes researchers come to me and I know that they care about these issues and want to see more research around it.

33:40 So when you look at the reward hacking and sycopency papers, you're not seeing, you know, focus on these particular issues. There could be a variety of reasons why I think it's undeniable that the kind of focus of AI safety has shifted it in one direction. If you look at like open AIs, like when they look at potential long-term risks, it's it's stuff that sounds a lot more sci-fi, right?

34:06 Or like is the subject of existing concern for effective altruists and the AI safety community, the rationalist community looking at risk from bioweapons, risk from super persuasion. So, like, you know, somebody using it to shift an election or something. They're not looking at these particular risks.

34:24 And to me, it just seems so obvious that subtle manipulation of our minds and our interior life and you know, the vulnerability that comes when you when you promote the use of the chatbot for therapy. The like, why, why isn't this issue prioritized? But again, I'm just, you know, this is me trying to understand why I'm seeing the papers that I'm seeing.

34:47 No, definitely. And I appreciate that. I I feel like me personally, I I've often found it kind of frustrating when we start to talk about these real issues, you know, with the chatbots. And before you can even get to talking to them, you need to kind of like diffuse and unpack these notions of the chatbot being sentient and trying to explain that that is not what's actually happening here, just to start to have like a baseline conversation about what's happening, which uh I feel like really kind of sets back any discussion of these real harms that we're talking about that can that can come of the way that these chatbots have been created and and manipulated and the way that the weights have been

35:22 set and things like that.

35:23 Yeah. And you know, some of it is just like jargon, which all researchers are prone to, right? But like reward hacking from user feedback and sycophancy, it just doesn't, you know, it doesn't bring that immediacy to mind. I will say, like Mike Krieger, who is the ex-co-founder of Instagram, who now works at Anthropic, he was recently on Hard Fork and he mentioned that there was an internal essay about why you shouldn't optimize for engagement.

35:51 And we haven't, in recent times, we haven't seen people showing these kinds of examples that they showed from OpenAI or from character AI from Anthropic. But you know, they didn't publish that essay. I would love to see it, but you know, I do think probably some of it has to do with companies also clamping down on publicizing research, you know, in a lot of cases because it's competitive, but I have to imagine in some cases also because it's controversial.

36:17 Yeah, no, that makes sense. And hey, if there's someone at Anthropic who has access to this, they can go ahead and leak it to Natasha.

36:22 Please send it to me. My signal is Natasha.10.

36:26 Now you know what to do. Yeah. Hopefully it arrives in your inbox soon. I I feel like, you know, when we talk about these things, you know, when I hear what you're describing, I think about this study that you cited in your piece from OpenAI that was talking about how people using these chatbots tend to be lonelier, have a greater emotional dependence on them, have lower socialization with other people.

36:48 And that also brings to mind these comments that Mark Zuckerberg was making recently about, you know, wanting to give people a load of AI friends because they don't have enough friends in their real life. And there's obviously something that the companies are doing here that they're responding to where it's like, yeah, we do live in a society where there is this degree of loneliness where it feels like certain people do not have, you know, the social connections that they would want to have or or that would feel adequate.

37:13 But on the flip side of that, it's like, are these tools really making that any better when they are, you know, trying to create these really intimate experiences that feel like, you know, they're almost tricking people as to the, you know, the type of thing that they're actually interacting with in some cases and whether it's really properly understanding with them and engaging with them and being a friend or a therapist or what have you.

37:36 Yeah, I don't know if you have any thoughts on the way that the companies are are approaching that.

37:40 Oh, yes, I have uh I mean, I would just advise everybody to listen to Mark Zuckerberg's interview on the Dwarkeshes podcast. It is wild to me, the portion where he's talking about using using these chatbots as friends. I mean, he talks about, well, first of all, let me let me back up and just say that like in talking to users, in paying a lot of attention to user forums, I see that people do find this extremely valuable, especially for use in mental health and loneliness and you know, in some cases for trauma.

38:16 Um, that company that I talked about, Chai, they used to have like on their pitch deck or on their promotional materials, like a testimony from a user who said they used it for eating disorders. And Chai has put like a bunch of its conversations online, and you can see there's a lot of eating disorder stuff in it.

38:32 There's a lot of other stuff as well. But you have people who are finding real utility in it. But I think if you're promoting that use case, you should also be testing that use case, right? And we're not seeing that, at least publicly. And so you have Mark Zuckerberg talking about how, like, he understands that people might find it off-putting, but you have to look at the quote unquote revealed preferences of users.

38:56 And it feels like code for, you know, he talks about the North Star of the data that that Meta is looking at. And it sounds like he's talking about optimizing for engagement. I mean, that's how other people read this interview as well. And he's just being really bold about it. And if you look, there's there's interviews from like back in 2019 or 2020 where he's talking about the challenges of this because Meta or Facebook then had been called out for using revealed preferences.

39:24 Like if you judge time spent on a conversation, you also have to judge like the entire power of Meta and every knob and dial that it can tweak to manipulate you to spend more time, you know, talking to a chatbot. I mean, he was just really bold about it, right? I mean, he made this comment about like most people have three friends, but demand for 15 more.

39:46 And he sounds like robotic, right? And he he kind of drew out this really dystopic scenario about like where you're basically just talking to chatbots all day. But to couple that with like our North Star is revealed preferences, I think it just shows you that it sounds a lot like what happened with the social media era.

40:05 I definitely agree with that. And there were a couple interviews he gave around that time that felt really like defiant, right? You know, that that we are going to do this thing that we think is good for the world, regardless of whether it is or not, because we can see all the other times when Meta was supposedly, or Facebook before it was supposedly doing things that were great and then ignoring the potential consequences that it just didn't want to see.

40:27 And it feels like once again, you know, Zuckerberg is doing a very similar thing with AI as he tries to capture this market and make sure that it has a good market share there. When you talk about these people who are using these products and these chatbots, you talked earlier about how some people are engaging with chatbots that are like anime characters and, you know, other characters in pop culture that they might like.

40:50 But then on the other hand, there are people engaging with these things where the AI is kind of positioned as being a companion or a friend or even a therapist. After you know, looking at the way that people talk about these chatbots, do you have an idea the degree to which people, you know, know that the technology they are engaging with isn't, you know, intelligent in the way that it might appear?

41:14 Or to a certain degree, do you feel like people are being misled about what they're actually engaging with and where their conversations are going?

41:22 Well, first of all, I would say the therapy, companionship, friend use case is often all in the anime chatbot. Like, you know, young people or people who like anime or like pop culture characters find it like a universal kind of feedback that you hear, at least from companies, they say that people feel a lot more comfortable talking to something that they perceive as a friend or a familiar character.

41:49 That's maybe those characters are getting more usage than the, you know, because they also have like AI therapist characters. So often it's one and the same. And we saw with another character AI lawsuit where this minor committed suicide, died by suicide, sorry, he was talking to the same chatbot, this like Daenerys Targaryen figure from Game of Thrones.

42:10 And sometimes it was therapy, and sometimes it was romance, and sometimes it was familial, but it was over time developing this closeness and this trust in the chatbot. And I I've seen some people say, like, well, it's just text. Like, what are you so worried about? But I think that you can see when you look at the screenshots, you know, you're talking to somebody that's not fully developed, right?

42:33 Like you're talking to them at the time that they are starting to learn how to have relationships and talk about these issues and talk about emotions. And, you know, so I would just say to the CEOs that are invoking the loneliness epidemic as a reason that their products should be marketed or pushed in this way.

42:54 I would just again say, like, are you testing for this? Are you seeing how they respond to deeply lonely users? And it's really fascinating because it, you know, there genuinely is a loneliness epidemic and there is like a lack of care and access to care. But you are having trillion-dollar companies go into this market in a way that at least we can all say there's no oversight, right?

43:20 Like these character AI lawsuits are the only reason that we're seeing these examples of vulnerable people being manipulated.

43:28 Yeah, I I just think it's so, you know, concerning to hear that. And on that point, like we just talked about Mark Zuckerberg. Do we see actions from whether it's the major companies like Meta, Google, OpenAI, or the smaller ones that you're talking about, you know, character AI, Chai? Obviously, replica was one that was getting a lot of attention for a while.

43:49 Do we see actions by these companies to actually try to address some of these issues? Or, you know, are they really trying to ignore that this is happening?

43:58 I think you can point to the OpenAI issue Study with MIT, the, you know, where they used actual human people, they tested them for four weeks as a sign that at least OpenAI is putting this research out there. It's the first of its kind that we've seen on this topic with humans, you know, recently. I know that Google, like part of the reason I brought up that Blake Lemoyne thing in terms of being able to play around a little bit with Lambda, is that Google just opened up its Gemini app to children.

44:27 So you see that it's been almost three years. They were definitely experimenting with it in some way back then. And only now are they releasing it to children. So I think that that shows that there's probably been, you know, some internal concern, right, about releasing it in a certain way. But I think also when you see adoption, like widespread adoption, like if you look at, you know, what are the most popular consumer apps?

44:53 It's often companions or AI homework help or, you know, chatbots in some way. So you have like companies that I'm sure would rather not be scrutinized and rather not be harming children at the same time facing a lot of commercial pressures. So it's only whistleblowers, it's only lawsuits, it's reports from consumers that are holding companies accountable.

45:17 There's there's nobody else looking at what they're doing behind the scenes to optimize.

45:22 Yeah, which is so often, I guess, what we see, right? Before there has to be this broader conversation, there are people who are looking into it to eventually try to force the companies to do something about it. You mentioned earlier in our conversation about business models, right? We know that these companies and their AI divisions are often losing a lot of money right now, you know, because of the costs of actually running these models, training these models, all these sorts of things.

45:46 Obviously, they are trying to get closer to profitability. What are we seeing in the business models applied to these chatbots? You know, the degree to which they're trying to implement advertising and other forms of monetization. And what impact do you think a greater push for profitability will have on, you know, the way that these tools are ultimately governed?

46:07 Well, I think we haven't seen anything yet. It's not, it's it's barely even been the first step. Like I saw that OpenAI executive recently said like he and Sam decided on $20 a month because it sounded like what they thought people would pay. So even with the pricing issue for subscriptions, because obviously that's another way that they can make money and are making money right now, and people are paying.

46:33 I mean, it, you know, they're losing money, but they're also kind of racking it up in the billions in terms of revenue. It's just ability to do that long term. But we haven't seen anything yet. There's not been an ad in one of these chatbots. But you you do see them hiring these people, right, who have specialized in this.

46:52 So I mean, just rewind back 20 years of Google and look at where Google is now. I think we are gonna see the exact same things, right? Like people are using these chatbots for search, they're using it as their portal to the web. So everything that we've seen happen to Google and to Facebook, I imagine we're going to see happen in chatbots.

47:17 Like a lot of these issues that I'm talking about, loneliness, mental health epidemic, like it's not happening anew with chatbots. It's happening when you have people who are already primed for this, right? Like they're already on their smartphone. And, you know, there's not even been enough studies to know like whether or not it is addiction or how, you know, sometimes it does help people.

47:39 It does make people feel less lonely. So into that morass, you are having like a bunch of little closed wall versions of these major companies. And in some cases, like in Google and Meta, it's the same companies, right? So I don't think we've seen anything yet.

47:57 Yeah, you know, the move fast and break things is not often the best place to be uh thinking about the safety and making sure that these tools are not going to have the harmful impacts that we're already seeing out there, and that I'm sure there are many things that we're not even seeing because not a ton of research is being done on it or not nearly enough, right?

48:15 Well, yeah, and I think we're seeing companies like I think it's a real test now of how much responsibility they want to take for the outputs. You know, you can see like Google and Character AI just move to have the lawsuit where I mentioned the kid cutting himself. They push to move it to arbitration.

48:32 So private arbitration, that means much less public scrutiny, right? And you see through the New York Times lawsuit recently of OpenAI for copyright infringement, they're asking them to keep their user chats. And that has helped bring more attention to the amount of data that they have kept on users.

48:52 So again, all of these issues around privacy and surveillance and how your own data is being used against you are, I feel like we're just at the start of it.

49:03 Yeah, the data question is such a big one that I feel like doesn't even get, you know, nearly enough consideration on the one hand, you know, have so much user data was already used to train these chatbots, but now how much they're they're keeping on people and you know going to be using to tailor, you know, the responses and things like that down the road.

49:21 But I I did have one final question for you, kind of picking up on some of the things that you were just saying there. You know, when you were saying that Google had opened its Gemini to children, I was struck by thinking about how in many countries right now, we're in the process of having this debate about actually raising the minimum age that kids and teenagers should be able to access social media at all, as there is this growing discussion about, you know, the consequences of social media that maybe we haven't been wanting to talk about for the past number of years, or, you know, we've even seen get worse because of these pressures toward commercialization and increasing the profits that

49:59 companies can extract from these platforms. I guess, you know, based on some of the things that I've been hearing you saying through the conversation, do you think that we're headed to a point where chatbots are going to start presenting this whole new range of concerns that are going to be similar to social media and that we're going to have to tackle anew at a time when we're just seemingly trying to kind of figure out what proper governance of social media should look like in the first place?

50:25 Yes, I think so. I mean, this has been the a theme throughout my work that I've been trying to highlight. You know, there hasn't been that much research on it and public data on it. But yes, I very much think we're going to be going through a lot of the same issues that we went through with social media, where it is a product manager in Mountain View, in Menlo Park, making a seemingly benign decision or, you know, a decision for commercial reasons that then will have impacts that we might not see for years to come in different markets, in different types of users.

51:02 You know, one thing about Google opening up Gemini to kids is the reason we found out about it is because they emailed parents who have like a Google link program where you have like some kind of parental controls. And on the one hand, you know, they're saying, like, hey, just FYI, next week we're opening it up to kids.

51:21 And I imagine that they want to do it for a AI homework usage because that's such a popular use case. Kids are already doing it. Would you rather have them do it with like an app that's owned by Chinese developers or Google that has more privacy restrictions in place, right? But if you read the email, they're saying like, please talk to your kids about how this is not really a human.

51:45 Please talk to your kids about how you can't trust the answers for that. And it just to me highlights like, you know, we don't even have enough digital literacy to be able to contend with the ways that social networks, you know, work on our human intuition and human instincts. And now we have chatbots like where is a parent supposed to go to figure out how to explain this to their child?

52:11 You know, I'm not seeing these same companies put out really helpful information into how the levers of power that they're pulling behind the scenes, the ways they train it, the ways it can go wrong outside. And at the same time, I'm not seeing media do a great job of that either, like explaining always how these things work, you know, partly because there is a lack of transparency.

52:34 But to me, yeah, it's just, it feels like we're speedrunning the last couple decades of tech.

52:40 Yeah, it's so grim. And even as you describe the contents of that email, like the contrast between how they're framing this rollout to children compared to how so many of the executives actually talk about these technologies in public and the huge difference that is in those two different framings of what generative AI is actually doing and what the chatbots actually are.

52:59 Natasha, keep up the great reporting. I really enjoyed this piece and having you back on the show. I think that this is going to be such an important topic, and I'm sure we'll have you back on the show to discuss it further as you continue, you know, kind of reporting on this. Thanks so much for taking the time to speak with me today.

53:14 Thanks for asking me. I I always love coming on the show.

53:19 Natasha Tiku is a technology reporter at the Washington Post. TechWon't Save Us is made in partnership with The Nation magazine and is hosted by me, Paris Marks. Production is by Kylie Hewson. TechWon't Save Us relies on the support of listeners like you to keep providing critical perspectives on the tech industry.

53:32 You can join hundreds of other supporters by going to patreon.com slash techwon'save us and making a pledge of your own. Thanks for listening. Make sure to come back next week.