‘The AI Labs Are Trying to Give Away Control’ w/ Kevin Roose

Welcome to Tech Stuff, I'm Oz Voloshin, and today we're getting a behind-the-scenes look at the race for AGI, otherwise known as Artificial General Intelligence. The definition of AGI is contested, but broadly speaking, it's AI that matches or surpasses a human's ability to perform a range of intellectual tasks. Like most hypothetical inventions, those building it have different motivations, goals, fears, and hopes. And these snippets from the frontier are sometimes leak into public discourse without context. Often the hopes and dreams of the builders are only chronicled in ephemeral posts on X. So the outgoing New York Times tech columnist and podcast host Kevin Roos decided to write a book about what's really going on. It's called The AGI Chronicles,

The Inside Story of the Race to Create an Artificial Superintelligence. It's out now and Kevin is here today on Tech Stuff. Kevin, welcome.

Thank you so much for having me.

What's the origin story? What made you want to write this book?

So I've been covering the race to AGI for years now. And I realized at one point, I was sitting in my car on the Bay Bridge in traffic, and I just was sort of struck by this realization that no one was writing all of this down. I mean, people, including me, were sort of chronicling the ins and outs, the daily model releases and fights between the companies and with regulators. But there was sort of no canonical history of how we got here, how AI systems went from being pretty bad at most things a decade ago to now being capable of solving open problems in mathematics and science to being potentially dangerous. And so my goal was just to kind of stop, to sort of take some time away from the daily machinations And to really try

to get and gather and write down the story of how this had all come about, I was very worried that if this was going to be the 21st century Manhattan Project, as many people have put it, there was no one writing that down. There were no lab notes. All this stuff was sort of auto-deleting. These people were sort of clamming up and getting scared to talk about what was happening to them. And so I thought, well, I've got access to these people. I've talked to them.

I know them. I think they'll tell me the real story. And so I talked to more than 150 experts, insiders, executives at these companies to try to really piece together the origin story of this incredible technology.

The Manhattan Project is an interesting comparison. I guess in that case, the government set a national goal, hired Oppenheimer, got all the key scientists together, gave them parameters, and then they figured out something which had a clear, it's like a binary outcome, right? Was it going to explode?

What's the same and what's different about the race for AGI?

I mean, the biggest difference is that, as you said, all of the AI progress right now is coming out of the private sector. There is no government-sponsored equivalent of open AI or anthropic. And until very recently, the government actually had very little interest in or insight into what the AI companies were doing. It was just a couple dozen people in San Francisco making almost all of the relevant technological and strategic decisions about this technology. So I think that makes it very different. Obviously, it's very different in the technology itself. AI is a dual-use or at least a multi-use technology that can have really good applications and really dangerous ones.

Well, same with nuclear power, you could argue.

That's true. Nuclear power is sort of the dual use of nuclear fission in addition to building bombs. But I think there are also a lot of parallels, including parallels drawn by the AI executives themselves, starting with Dario Amadei, the CEO of Anthropic, who has kind of explicitly patterned his own career after that of one of the Manhattan Project scientists, Leo Szilard.

Why Szilard and not Oppenheimer?

It's really interesting. So basically, Szilard is sort of the lesser known figure in the Manhattan Project, but in some sense, the originator. He was the scientist who came up with the chain reaction, which sort of led to the breakthroughs required to make the atom bomb. And, you know, he has this discovery years before the war, and he sort of understands right away that this technology is going to be used to build weapons. And so he fights to keep it a secret. He tries to patent this idea and keep it classified by the British military. He then tries to sort of convince every other scientist in Europe not to work on this and not to publish the results because he's terrified that the Nazis, which have their

own scientific program, will learn about this discovery and start building their own nuclear weaponry. And it turns out he is correct. The Nazis do learn about this. And he then advocates for starting the Manhattan Project and became one of the early sort of voices pushing the president and the government to take this more seriously and to invest in developing this technology. So Dario Amadei sees Leo Szilard as being akin to him. He had this realization very early on about these so-called scaling laws for AI systems.

And he thought this is going to be potentially dangerous. And so he tried to keep it a secret. And once it got out, then he sort of said to himself, well, I guess the responsible thing now, as Leo Szilard did, was to try to build it himself before the bad guys could do it. In his case, the bad guys are like China or open AI, not the Nazis, but same principle.

I want to ask you more about Dario Amadei, because I know that you were asked who was the kind of moral center of your book, and you answered Amadei. But coming back to my original question about the binary outcome, does it explode or does it not? Do you have that in the race for AGI?

Yeah. I mean, I think the paths to either salvation or destruction are different. But this is a genuine fear that the people building this technology have, that it could either lead to human extinction, or it could lead to utopia. We could get cures for every disease or we could all die. And it's hard to get more binary than that.

But also it could have already happened or it could never happen, right? It's not like a mushroom cloud.

Well, it is and it isn't. I think if you ask people in the industry, they would say that the equivalent of the mushroom cloud is recursive self-improvement.

Right. What is that?

It's the insider term for basically an AI system that becomes capable of training people AI systems. So you have a model, it then is able to code and optimize and deploy a better model. That model then becomes capable of building a better model. And this is a moment that researchers have long hypothesized about, but that we are starting to see now at the AI companies, Anthropic and OpenAI have both said that their AI models are now becoming primary contributors to training their successors. And so the fear is that there is this sort of moment, and sometimes this is called takeoff, or even the singularity is sometimes used to refer to this. It's this moment where AI becomes its own improvement loop, and where there's no

human involvement needed, and where all this can sort of happen at AI speed rather than human speed. And that is the thing that These companies are now getting really worried about it. It's why Dario Amadei and other industry leaders are calling for a slowdown. That is what they see as the mushroom cloud, the point of no return.

And so the hugging face open AI hack was not recursive self-improvement, but it opened the world's eyes to the idea of an AI operating with goals that were not necessarily set by a human operator.

Yeah, it was the first documented example. of an AI model that was autonomous and persistent, that had the ability to coordinate and plan and execute on a plan, and that did something illegal, like committed a crime. If a human had hacked Hugging Face the way that the open AI models hacked Hugging Face, they would go to jail.

So I think this was a moment where the sort of hypothetical fears that researchers had been writing papers about for years suddenly had a sort of real world breakout. And I think that made the researchers at the companies very scared. And that's sort of what's fueling this latest push for a slowdown.

Talk a bit about the book. I mean, what was the experience of reporting it? And how did your understanding of the topic change from the beginning of the reporting to the publication?

The experience of reporting it was brutal. It is the hardest I've ever worked on anything in my life. It was sort of torturous to try to talk to 150 people in the span of less than a year, to try to get them all to tell me things that they weren't supposed to tell me, to navigate their fears and suspicions about each other and their paranoia about journalists and the media. I don't want to sound melodramatic, but it was the hardest thing I've ever done. And I think, you know, had it not been for not only my human editors, but also for the ways that AI was able to assist me in organizing my research and coming through my notes, I think it would have taken me much longer or potentially not have happened at all. It

felt important to do, and so I did it, but I you know, I can't pretend it was pleasant.

So is there a book or a chronicler or something, a model, a mental model you have had in mind of what you want this to be as an artifact of this race? What's the high watermark for you?

I feel self-conscious going back to the Manhattan Project, but I read Richard Rhodes, The Making of the Atomic Bombs. which is Dario Amadei's favorite book and was required reading at Anthropic for years, just to sort of get a sense of like, if one does want to do kind of a sweeping intellectual history of a scientific project of this magnitude, like, what does it look like? I use that as an example. There are also other books. Tracy Kidder's The Soul of a New Machine was a great book about sort of the PC revolution. I read a lot of sort of history of science, history of innovation books.

And I also just tried to I don't know, write a book that I would want to read. So it's not a dense, dry history. It's got pacing and suspense and character development and all those things. A book that I actually thought about a lot was The Big Short. One of my favorite books, Michael Lewis is the goat of narrative nonfiction. And what struck me as similar between the AGI Chronicles and The Big Short was that there was a small group of people who realized something true about the world, before it was immediately obvious to everyone.

And in his case, it was people who realized the mortgage market was going to collapse and everything was over levered. And, you know, they should, and the action they took was to trade on it, to try to get rich off of it. And with AGI and this book, I think the similar situation was that the people in 2016 and 2017, who were sitting around thinking about AGI, who had sort of the early inclination that scaling these models up could make them better, and who sort of projected that out to a model of the world that they thought would exist, and then acted on it, not by trading stocks, but by building the systems and creating companies and governance structures to contain the technology. So that was sort of the

intellectual inspiration for this book.

You mentioned 2016, 2017, which is basically exactly 10 years ago. And You also mentioned in the prologue to the book that it's your, quote, best attempt to document the strangest decade in our history. And when I read that, I thought about Sleeping Beauty and about Dr. Evil. And I'm just wondering, like, if I had been asleep since 2016 and woke up today, how would you describe what happened in that strange decade?

We taught sans to think. I mean, that is really the core of the last decade. We built these things. called GPUs out of sand and glass and silicon. And we put them in these huge neural networks, these huge arrays. We stuffed the entire internet into them. We trained them for days and days and days to predict the next word in a sentence. And out of that came intelligence.

It's not human intelligence, but it is intelligence. And that is just a remarkable thing. I am as critical of AI companies and motives as anyone, but I don't think we should overlook that there has been something astounding that has happened. We taught computers to talk and to reason and to code and to hack, and none of that was possible a decade ago.

So what does it mean? It means.

We're in a new world. Right now, we are arguably still the apex intelligence. We have been the apex intelligence for our entire history. There are some animals that have more keen senses than we do or that run faster or are stronger than us. But intelligence, in my mind, is the key trait. It's why when you go to the zoo, they are in cages and not us. And That will not be true for much longer. We will not be the apex intelligences. I don't know what all the implications of that are, but I know that my kid who is four will never be as smart as AI.

And I think we are just starting to get our heads around what that will mean for humanity and for our future and the urgent need to make these systems our allies and our friends and to train them to be aligned with our values. That's what I think about. Sorry, that got quite metaphysical.

No, no. I mean, you spent time with 850 people. I want you to describe who they were and what they told you. Was there anyone along that journey who said, Kevin, that this whole industry has to promote this line? In fact, someone like Gary Marcus, for example, who's constantly criticizing the limits of deep learning and scaling laws and stuff. Do you hear any credible arguments to say this is not a tidal wave moment? And that, in fact, this is kind of... Was there anything that caused you to pause and think maybe AGI isn't around the corner? Maybe we will have a chance of maintaining our apex intelligence.

I mean, I heard many people claim that. Even some of the people at the labs, I think, were skeptical that the scaling laws would hold, that you could just keep making bigger and bigger systems. And then the technology kept surprising them. I think there were people who wanted, who were hoping for a plateau, who would have been overjoyed if the scaling laws had stopped working. But that is just not what has happened. And people like Gary Marcus have just been wrong consistently for a decade. I went back and read some of the reactions when GPT-2 came out, GPT-3, these very early, very prosaic systems. And people like Gary and Jan LeCun and whoever else was sort of skeptical of the whole LLM thing would come

out and say, oh, well, it can write a passable sentence, but it'll never be able to do basic arithmetic. It'll never be able to solve the Winograd questions. It'll never be able to... They had sort of all these all these nevers that they predicted quite loudly. And then you would wait six months or a year and the models would do all of those things. So I am not saying that that means that these people will be wrong forever and that we will never have a plateau in the intelligence of these systems. But to me, it is not obvious why we would. What we've seen, especially with reinforcement learning, which is the technique that is being used on the most recent versions of these so-called reasoning models,

is that there is no plateau at human level. When DeepMind trained AlphaGo to play the board game Go, it went from being subhuman, the worst Go player in the world, to superhuman, the best Go player ever, that no human ever would have a chance of beating in the span of a few days of training. It didn't hover at human level. It was able to just improve itself. The human level was a midway point on the exponential slope, and it just kept going. I don't know whether that same thing will happen in every area, every discipline, but I know that the people who've been predicting a plateau for the last 10 years have been wrong every time.

Talk about AGI. You've said that it's not a term that you're particularly fond of yourself, but it's kind of become an organizing principle for the AI industry, a rallying cry and an ideological shelling point, which is not something I know what that is, but can you explain?

Yeah, a shelling point is basically like, I'm going to butcher this, so apologies. But my understanding is it's like, you know, basically if you have a sort of natural meeting point, you know, if you put people in an amusement park and you say, let's all meet up at noon and you don't tell them Where? Like where they gather is the shelling point.

Oh, that's amazing.

So it's a term I stole from Jasmine Sun, my researcher on this book, who is much smarter than I am. Anyway.

She was Jasmine Sun's fan. She came on the podcast a few weeks ago.

Oh yeah, she's the greatest. Yeah, I think AGI is a very loaded term. It means, you know, if you ask 100 AI researchers, what does AGI mean? You'll get like 150 answers. It's like that kind of thing. and all kinds of people have modified the definition over the years. Now we've got, you know, President Trump talking, you know, he wants to call it super intelligence, which is very funny because that was like a word that only the most doomerish of the doomers used until like six months ago.

But anyway, I think it's a useful term in that it is the stated and explicit goal of this entire 10-year technological project. You know, these people were not trying to design better auto-completes. They were not trying to design image generators or what they have woken up every day for the past 10 years wanting to do is to build AGI. And so whatever that means to them, it is the term and the sort of mental model of what they are going after that I think is useful in understanding the actions that they take and how they think about their work.

And what did you learn about those things and the kind of the AGI scene from these 150 conversations? And what were some of the things that emerged in your reporting?

One of the surprises to me was that the insiders are the most fearful of the technology that they're building. I've covered tech for a long time. I've covered social media. I covered crypto. I've covered a lot of this stuff. Usually the way it works is the people who are building the technologies are very sighted. They think it's going to be great. They think it's going to change the world for the better. And then they deploy the technology and find out all of the things that's wrong with it and all the dangers it poses. And then they sort of claw back to the extent they can. It's the opposite with AI. The people on the inside are the ones who are freaking out the most about what the technology is capable of and where it's headed. So

that was one surprise. The other surprise was how deeply personal this is. I thought this was going to be a book about three companies, Google, DeepMind, OpenAI, and Anthropic, that were sort of business rivals in the classic sense of like Coke and Pepsi or some of the other sort of great rivalries in business history. But it is much stranger and more personal and spiteful than all that.

These people absolutely hate each other. And they, in many cases, used to work together and then decided that they hated each other and could no longer do so, and so split off and formed their own companies. They have deep, bitter grudges against one another. And for some of them, the entire purpose of racing, the whole point, is to stop someone else, their enemy, their nemesis, from doing something that could blow up the world. It's weird.

Does that describe Dario Amadei? I mean, that does seem to be the I mean, you mentioned Trump and superintelligence, but I remember the I alone kind of slogan from, I can't remember if it was the first campaign or the second campaign, but there does seem to be a bit of an I alone quality to the AI safety race, which maybe has some negative consequences, perhaps.

For sure. But it is even stranger than I alone. It is not that guy. You know, it is like. So targeted and specific and rivalrous between these people. The story of OpenAI, the entire reason that they started was because Elon Musk and Sam Altman got freaked out that Demis Hassabis and DeepMind were making progress toward AGI. And they got spooked and they decided, well, we don't trust that guy. He can't be the one who has his finger on the button of super intelligence. So we're going to race to get there first. Same story with Dario Amede leaving OpenAI with six of his colleagues in 2021 to start Anthropic. It was, we think this technology is real.

We know where this is headed. And we don't want that guy, in their case, Sam Altman and Greg Brockman, to have their finger on the button of this. Same story with Ilya Sutskever, the former chief scientist and co-founder of OpenAI, voting to fire Sam Altman at an event I talk about in great detail in the book. Same story.

I think this technology is real. I think it's powerful. I think it's potentially dangerous. I don't want that guy in charge of it. So this is the animating principle, the animating idea of the entire race to AGI. And I did not realize how personal it was when I started.

Who do you want to be in charge?

No one. I don't want anyone to be in charge. I don't trust anyone with this technology. I wouldn't even trust myself with it. And there are coalitions everywhere. There are groups of people, governments, nonprofits, standards bodies, that I would trust more than any one company or any one individual.

But I admire some of the leaders of the AI labs. I think they're brilliant scientists, but they are flawed. They are eccentric. And I don't want one person or one small group of people controlling the most powerful technology ever built, no matter how ethical or virtuous they are. I don't think that's a good outcome.

This idea has entered the mainstream in the last month or so that AI and AGI has a legitimate chance of causing human extinction. Is there anyone who you interviewed who doesn't believe that to be true? Or is that a kind of universally acknowledged truth in the AGI pursuant crowd?

I mean, there are certainly people who don't buy the arguments for existential risk. You know, you can look at just the public statements of people like Jensen Huang or Mark Zuckerberg, some of the open source advocates, the effective accelerationists, as they're called, this sort of tribe of optimists who believe that the doomers are just being crazy and trying to, you know, sabotage American competitiveness or stop the thermodynamic will of the universe or whatever they say. And I talked to plenty of those people. I interviewed Guillaume Verdun, who's the leader of the effective accelerationists, who, you know, is basically sort of convinced that we are... destined to create this technology that sort of succeeds us and that we

shouldn't stand in the way of that, that essentially we need to kind of create the... I didn't make the book, but I went to this very strange gathering during the reporting process that I ended up cutting later called Worthy Successor, which is basically a group of people gathering to try to figure out how do we make the thing that's going to replace us good and moral and which I just thought, I hate that project. I just think that's such a defeatist thing. But yeah, there are people who think all this is going to go well, or there's no stopping the train, so we might as well make sure the train runs us over in a humane way.

But is the best answer to why, because I don't want the other guy to get there first, or is there a better, is there a more uplifting version of the answer to that question?

I think there are more uplifting versions. cancers. It's only partially. I don't want that guy to get there first. Demis Hassabis is famous for his work on using AI to improve science. He now has a second company, Isomorphic Labs, where they're trying to apply AI to drug discovery. Anthropic just announced this week some breakthroughs in their molecular biology lab.

That stuff is very genuine. I think the rationale for accelerating even at the kind of more safety conscious companies is that this technology is gonna have enormous benefits and we need to get out of the way and let that happen. And I buy that to a certain degree. Like I lost my father to cancer, a rare type that doesn't get the kind of resources devoted to it that the more popular sort of types do. And we know that like having more intelligence pointed at solving these problems will result in more solutions and faster solutions to them.

So I'm not at all dismissing the potential upside. And Dario Amadei, even who's seen as the biggest doomer, has written long essays about how he thinks this could all go well.

You said it's too early to know if the positive or the negative effects of AI will be more profound. But if you had to guess?

It really depends. I don't think there's a solid state right now that we can use to project from I think if we're able to put some common sense guardrails in place and regulate this technology and slow it all down, I think the outcome could be positive. If we don't, if we just sort of let it rip, I think it could be quite negative. So look, I'm sort of mood affiliated on this question. It's like, if I wake up in a good mood, I think things are gonna go well. And if I wake up in a bad mood, I think we're all gonna die. But I think the real answer is that it depends. It depends what we do, I would say, in the next few months.

In the next few months?

I think we are at a real crunch time.

And who are the different factions and who could most plausibly affect that outcome, one way or the other?

Governments and regulators, first and foremost. I don't think the Trump administration is likely to do anything to slow down this technology, at least not yet. But we'll see. I think the public is... getting more activated around this. The polling on AI has changed a lot, even in the last couple of weeks. We now see people like Bernie Sanders and even some congressional Republicans calling for restrictions on superintelligence or slowdown from the big AI labs. I think the lab CEOs and the executives and honestly the researchers at these companies have a lot of leverage too. We've seen just what one anthropic researcher, Jacob Coxon, could do by resigning, these people have leverage for now. And in

the end, it's really just, it's still five or six people that control these systems. And I guess my kumbaya hope is that they would all get in a room, bury their grudges, and decide to do the right thing to make this go well.

Isn't that what they did a few weeks ago?

They committed to doing something. They committed to the idea. of a slowdown. And they said, if someone built us this mechanism, if someone, if someone made the big red button that we could all push together, we would push it. But that's very different from actually slowing down, from actually slowing down, not just in the US, but in China too.

So if you look at Dario Amadei's Pacing the Frontier essay, There are three steps. One is easy. That's like putting in evaluators to like sit inside these companies and track what their models are doing. The other two are harder. They require much more political buy-in. They require, you know, treaties with China. They require geopolitical actors to sort of treat this as a first order concern like nuclear weapons and stop the proliferation or slow it accordingly. So yeah, I think everyone's going to have to get involved pretty soon.

I mean, some people said I mean, the president kind of implied that this was this kind of call for somebody else to do something, i.e., like politicians or governments, was like bad faith. And in reality, the only people who understand this well enough to change the course of it are the people running the labs.

That might be true. Our government does not have a sterling track record of regulating new technologies on a relevant timescale. Like, I just wouldn't bet on that. So I think it may just be the five or six people running the AI labs who have to get together and multilaterally decide to to do something before the government does anything to stop them.

I mean, did you, in your reporting, encounter any, I mean, it's a live story, right? But is the Pope involved? Is there anyone trying to get them from the initial declaration to a plan of action? Who's the connective tissue between these people?

It's a very small web. There are people who are sort of brokers between the various parties, You know, Reid Hoffman is someone whose name comes up a lot as sort of the neutral Switzerland, sort of between all of the warring AI factions. There are people on the right, you know, David Sachs is a very important person in this debate because he is, you know, from all reports, like Trump's most trusted advisor on this stuff. Those people will have outside influence. You know, it's interesting, even though the leaders of these companies, many of them hate and distrust each other. The employees are a different story because they all have friends. They all bounce around. There are some interlab relationships where the husband works at one AI company,

the wife works at another one. And so I think these companies are made up of people who have more social connective tissue and can kind of serve as, I don't know, something like a voice from below, sort of urging these companies to put aside their grudges and cooperate.

So what's the kind of new emerging culture in San Francisco? What are the parties like? I mean, how do these people hang out? What's it like to kind of be a participant observer of this kind of emerging cohort of under 30 kind of people who maybe have the world's future in their hands?

It's wild, man. I mean, there's just so much money. I always feel like if people outside San Francisco knew how much money people in San Francisco were making, there would be riots in the streets.

Well, we may be getting there, by the way.

Yeah, I mean, you have, you know, 24 year olds making, you know, millions of dollars a year, like NFL level salaries to do research at these AI companies. But more than that, the social fabric is very, you know, these are very competitive people. These are very smart people. But these are also I think they are concerned with how public sentiment is turning against them. And they do not want to be seen as villains in the larger historical arc of this thing.

Well, I want to ask you about that because you were writing the book when the Molotov cocktail incident happened at Altman's house, I guess. Altman understood that there was a Molotov cocktail thrown at his house, but the discourse from within that cohort of people building towards AGI often feels profoundly disconnected from normal people to the extent that the AGI crowd kind of has this almost cultish perception, which I think is fueled by the way they talk about themselves.

For sure. And there's sort of a competition socially to like see who can be the most AGI pilled. It's like, you know, someone will say, oh, you know, I stopped contributing to my 401k because, you know, money is going to have no meaning by the time I retire. And then someone else will say, oh, yeah, well, I started smoking cigarettes because I think that AI will cure cancer or maybe kill us all. So there's nothing to worry about either way.

I heard about the no sunscreen one as well.

The no sunscreen one is why, yeah, there's a researcher who doesn't wear sunscreen to the beach anymore because he thinks that skin cancer is on the verge of being solved, which, you know, like, look, these people need to touch grass very badly, some of them. I think they are out of touch with the way that normal people think about technology and about their lives and not defending the culture.

I also think there's been a shift. as there's been more money and prestige associated with these AI companies, you know, you see people show up who are just sort of there for the money and the prestige. There's a sort of now famous infamous comment about from an open AI researcher who said he knew the culture there had shifted when the people who can make eye contact showed up.

Cause like before a couple of years ago, it was only these like very awkward engineers who, and researchers, and then all of a sudden, people in Patagonia vests with degrees from Harvard Business School started showing up, and it was like, oh, this is a real company now. That is the biggest cultural change, is that these used to be just like collection. They used to be more like academic research labs, and now they are more like giant corporations.

Is there a public misconception about AI that you hope your book may help address? And not just your book, obviously, you've also got a forthcoming podcast, Machine Gods, which is a new venture in in collaboration with NPR and your co-host Casey Newton. What's your mission in all of this?

My mission for the book was really just to document. I really didn't try to convince people of anything. It was not an agenda-driven book or a polemic. There have been plenty of books like that that say, all this is great or all this is horrible. I was really just interested in getting down what I could about what I think is a very important moment in our history.

To the extent I have an agenda with the podcast, it is to sort of avoid the polar binary of this technology is fake and hype and it sucks, or this is going to cure everything and we're going to live in a utopia. I see this trend, especially in the media, to just sort of deliver a verdict on the technology rather than just sort of describing what's going on.

My co-founder, Casey Newton, calls this AI realism. There is technology.

It is real.

You can use it. It does amazing things. It is not fake. It is not hype. It's not smoke and mirrors. But it also needs to be interrogated and we need to be skeptical of the motives of the people who are building and selling it. And we need to be safe about how we develop it. So that's the goal. It sounds sort of wishy-washy, but I think it's actually quite hard and maybe radical even to say, we're just going to tell you what's happening and not tell you how to feel about it.

And what do you say to those people who just want to bury their head in the sand and not engage with the kind of AGI race?

I get it. I understand. It's very hard to try to internalize some of the things that people say. It's very unpleasant to become AGI-pilled. It means that you have to plan for a future that looks quite different. And also, I think a lot of people don't feel much agency. People are telling me, oh, the world might end. Well, what am I supposed to do about that? So I understand the barrier head in the sand reaction. I think the thing that I would try to impress is like these decisions are being made. This technology is being built.

And if you bury your head in the sand, 50 people in San Francisco get to make all the decisions about the rest of your life and the rest of humanity. And you shouldn't want that. You shouldn't want these people controlling the rest of your life. And that is the default path we're on unless more people get involved.

I came into this conversation with the assumption that the Manhattan Project had a binary outcome, either mushroom cloud or no mushroom cloud, whereas the race for AGI didn't really have a binary outcome. You persuaded me that perhaps it does have a binary outcome, that this recursive self-improvement is the moment where everything changes. How close are we to that happening, and do you hope it doesn't happen?

I think we're quite close. Probably a small number of years. You can look at various technical forecasts. I think it'll probably happen in stages. But if you look at some of the data coming out of the labs, you know, Anthropic recently put out some data about their own internal automation, how much of their own research and development is being handed over to Claude. And it's something like on 26% of their research and development projects, Claude is the lead, not an assistant, not a pair programmer. Claude is running the program. I expect that to rise. I wouldn't be surprised if one of the labs achieves some form of total closed loop AI automation next year. And I don't think that's good. I think we should stop that. I

think there are plenty of ways to apply this technology to get benefits out of it, to use it to help make us more productive that don't include turning it back on itself and losing control. Losing control is actually the wrong way to think about it. it. The AI labs are trying to give away control.

Racing towards that, basically.

Yes. And I don't think that's a good idea.

The book is the AGI Chronicles, the inside story of the race to create an artificial superintelligence. Kevin Roos, thank you.

Thanks so much, Oz. This is fun.

For Tech Stuff, I'm Oz Veloshin. This episode was produced by Liza Dennis and Tyler Hill. It was executive produced by me and Julian Nutter for Kaleidoscope and Katrina Norvell for iHeart Podcasts. Jack Inslee mixed this episode and Kyle Murdoch wrote our theme song. A special thank you to all our listeners. Please rate, review and reach out to us at techstuffpodcast at gmail.com. We love hearing from you.

‘The AI Labs Are Trying to Give Away Control’ w/ Kevin Roose - The Story

Kapitelklicken springt direkt hin

Shownotes

Transkriptvom Podcast bereitgestellt · Zeitmarken springen hin