Garrison Lovely on the case for banning superintelligence

0:00 There's this tiny group of people that's trying to render us obsolete. Almost none of us want that to happen, but we're letting it happen because we either aren't aware that it's happening, or because we don't think they'll succeed, or because we don't think we can stop them.

0:20 Recent weeks have had some news that's raised some of the big questions about where AI is going. We had the hugging face attack, we had some big mathematics results, and we've had some People resign from the AI labs, warning about the possibility of existential risk. It's the perfect time for a book about the big questions in AI.

0:39 Today, I'm thrilled to be joined by Garrison Lovely, who's the author of a book called Obsolete, which is about what he calls the obsoleting project, which is the AI industry's efforts to build AI systems that can replace human workers and transform the economy. I would say he's not a huge fan of this project and thrilled to talk to him about it.

0:59 So Garrison, welcome to AI Summer. Yeah, thanks for having me. So I feel like you've been somebody who's been thinking about these issues for quite a while. I started writing about AI full time in 2023. I felt at that point you were already pretty actively involved in these issues. How did you get interested in these issues and what's been your kind of role in the debate before you decided to write a book?

1:18 Yeah, I was a lurker for a long time. I first read about AGI in a Wait But Why blog post series back in 2015, 2016, which was itself just summarizing mostly Superintelligence by Nick Bostrom, which I also read around that time. And my feeling was like big if true, like conditional on AGI being possible.

1:38 I get why that would be the biggest thing. Thing in human history. I was just pretty skeptical about when it would happen. And I kind of looked askance at the people trying to do math to figure out how to make this go well when there were like these really big problems we were experiencing here and now.

1:54 But I kept, you know, tabs on it. I read books every few years about it. And then I got into journalism independently writing about my experience at McKinsey working for Rikers Island and ICE and kind of seeing how These elite corporations can end up abetting what I saw as humanitarian catastrophes in waiting.

2:16 And then after Chachi B.T. came out, I saw... An opportunity to write about this because previously I was like, eh, I'll leave AI for the experts, the people who actually know linear algebra or something. What can I really add? And then after AI became a bigger thing with JGBT, the average article maybe went down in quality a bit or maybe I was thinking too highly of them before.

2:40 I was like, I can maybe bring up the average. I realized that there was this kind of wide open lane looking at the political economy of AI and specifically AGI, like the thing the industry is trying to build. I wrote this piece that became a cover story for AI. For Jacobin Magazine, where I interviewed experts from the three major camps on the AI X-Risk debates, the kind of safety people, the ethics people, and the boosters.

3:04 And then that resulted in a book deal and the book that we're talking about now. And in the intervening years, I spent a few months covering SB 1047, this California AI safety bill, which was the first major regulatory fight in the country over the question of like AI catastrophic risk. And also covered a lot of opening eyes restructuring for mySubstack and was just building up.

3:31 I think it was helpful to be doing this more immediate reporting on the industry because writing a book, you could be Very removed. You could think about the abstract, but I felt like these specific nuts and bolts fights over how this technology would be governed, if it would be governed by anyone besides the people building it, ended up being quite relevant to the perspective of the book, which is many things.

3:56 But one of the big ones is that this is a question of political will, and we have to overcome an incredibly powerful and wealthy industry to get good outcomes from this technology. At least that's my view.

4:08 Yeah. So you mentioned that you wrote for Jackman. I believe you've also written for The Nation. And so you have connections on the left-hand side of the political spectrum, and especially like not liberals, but like actual leftists. And the early part of the book, you actually spend a fair amount of time, I I think in the kind of intramural arguments with other leftists who are very dismissive of this technology, you criticize the stochastic parrots point of view that you have some academics that this is making kind of theoretical arguments that these models are not on path to reach human level intelligence.

4:36 You also write a fair amount about Ed Zitron, who is a PR person, but has recently become a well-known figure arguing the kind of financials of this industry aren't going to work. Talk to me about how you see the ideological landscape there. And I'm particularly curious. How big do you see the faction on the left that does take this seriously and kind of sees things the way you do?

4:57 Do you feel like you are a little bit of a lone voice or is there like a lot of people on the left who see issues similar to the way you do?

5:06 Yeah, I think I was early, but I was sort of like a lone voice in the journalism intersection with like AGI might be real world who's also on the left. Yeah. There definitely has been a faction of the left that takes AI seriously and sees it as a potential tool for liberation and something that could be socialized.

5:30 This is a view that's been expressed in Jacobin previously by people like Nathan Robinson and Matt Brunig and Luke Savage. I think that could end up becoming the big cleavage point. If AI is a big deal, it becomes a dominant view, which I think capabilities progress will make it harder and harder for that not to be the case.

5:53 I think the big cleavage point will be between people who want to just stop it and those who want to nationalize it and use it to fund an incredible welfare state. I am quite worried about the alignment piece of that, which we could get into. The question of How prominent of a view is mine versus the stochastic parrots or the bubble view?

6:17 If you look at commentators, I think, and a lot of the commenters on YouTube videos about this, like Whenever I go on Breaking Points, there's a lot of people complaining about me and a lot of people being like, bring real experts like Ed Zitron on. And so there are very, very big fervent fan bases for his point of view.

6:38 And I think the kind of AI is just marketing hype perspective is the dominant train on at least like the portion of the left that comments on Blue Sky and Twitter. If you look at surveys like polling of Americans and bisected by their political views, like The risks are quite real to them. And I worry that there's this like chattering class that's sort of speaking for a silent majority potentially of people who are quite concerned about this.

7:07 I've also seen a lot of movement in the last few months, especially in the wake of Hugging Face and the first few days after Jacob Coxon's resignation, I was seeing people come around to the idea that AI might be a big deal and it might be something like Adam Johnson from Citations Needed saying like AI doom as in AI kills everybody.

7:28 It was like silly and sci-fi, but AI doom as in like Humans lose control of AI sometimes and it causes a bunch of harm to people. It's obviously real. And I'm like, cool, great. And people are going to metabolize this in their own ways. And I think that should be encouraged. And I think it's totally fair to still be skeptical of the more necessarily speculative thing of the full-blown human extinction stuff.

7:50 And a big point I try to make in the book is you don't need to believe that at all to get to the same position, which is we should stop trying to build... Universal labor replacing machines, what the industry calls AGI. That looks like a good idea under many, many different worldviews.

8:05 Yeah. And how important do you think Bernie Sanders has been in this? I mean, he has become much more vocal about this issue in the last six months to a year. How much has that changed left of center thinking about these issues?

8:17 Yeah, I think it helps a lot. The people who are still skeptical, the prominent ones like Zitron and Emily Bender, they're like, oh, Bernie's lost the plot here. They'll say the same things. I don't think they're moved by it. But Bernie is the most popular politician in the country after Obama. And so I think it's incredibly helpful for You know, normalizing, taking the risk seriously.

8:39 And I think most leftists, I mean, there's some people who are mad at Bernie about Israel or any host of things because like people on the left like to be mad about various things and, you know, with various degrees of justification. But I think he's still quite well regarded. And I think that It's been helpful.

8:59 And within the kind of left wing of the Democratic Party in elected office, AOC has signed on to the banned superintelligence bill. Greg Kassar, who's the chair of the Progressive Caucus in the House, has also I've been one of the co-sponsors of that with Bernie. And so I think it's notable that people who are elected to represent people and are looking at polls and actually are accountable to the public in some way are embracing these kinds of policies which are seen by people in the Emily Bender, Stochastic Parrots world as like They're calling it regulatory capture or totally unnecessary or something.

9:39 And so I think those perspectives are going to be increasingly marginalized, but we're really less prepared than we could have been because that perspective has been dominant for so long.

9:50 Yeah, no, I definitely think you're right that that view is not going to age well and that, you know, it's like the internet is no more important than the fax machine type view in the 1990s. You didn't really have to argue against it. You just had to wait and like... It was going to become obvious that that was wrong.

10:16 And in particular, I think you draw distinctions between different kinds of AI and think there is maybe a more positive future than involves some AI. How do you think about, yeah, what's the bad we want to avoid and what's the good future we can maybe steer towards?

10:30 Yeah. So the thesis of the book is there's this tiny group of people that's trying to render us obsolete. Almost none of us want that to happen, but we're letting it happen because we either aren't aware that it's happening or because we don't think they'll succeed or because we don't think we can stop them.

10:48 And the book identifies that this is happening, explains why it could succeed and why the only surefire way works. It won't succeed as if we do stop them. And I think the obsoleting project is a useful frame because we think of AI now as the kind of AI that the industry is trying to build, which is like these chatbots and now AI agents, maybe generative systems like Suno and these like video models that are designed to kind of replace human labor and human creation.

11:17 That's not the only type of AI that we could have. And I think it would be great if we had more things like AlphaFold, which cracked the protein folding problem and unlocked a bunch of benefits for medical science. And it's the type of thing that AI is really good at. And it's interesting that Google scrapped that team a few months ago and rolled it into Gemini.

11:40 And my argument is like, We're going to keep getting the obsoleting project because that's what's making money and they're going to start with the high margin careers like AI, R&D, investment banking, consulting. Engineering, software engineering. And we're not going to get nearly as much of the amazing medicine that they promise because that's just not where the money is.

12:03 And we could just go straight for the cancer cures and the new drugs and the material science stuff. And we could do that with neurosystems. And avoid the biggest risks. And also avoid this problem of what do you do if all white-collar remote work is replaced by machines in a matter of years on the path to physical work following with robots.

12:29 And that, I think, does carry big risks, including extinction, on a long enough timescale, which is still not that long in my mind. But also just the default best-case scenario is that the Machines are superhuman across the board, and then they do what they're told, and then they empower whoever controls the machines.

12:50 And this will disempower workers and create incredible amounts of power concentration in the companies building them and the governments controlling those companies. And the people who are in power are not going to suddenly become enlightened by having access to superintelligence. They're going to, by default, use it to do what they're already trying to do, which in many cases involves ethnic cleansing and other horrors that we should oppose.

13:15 And so I think the status quo of letting this happen is like, yeah, our default is doom or dystopia, and which we get really depends on what the machines do and whether they do as they're told, which is a big open question right now.

13:29 What's the role of robots in this? You mentioned robots a second ago, and it seems like if you're thinking about it as a project to replace labor, most labor involves physical work. We don't currently have robots that are, I think, very close to being able to do most physical labor. Does the bad outcomes that you're envisioning, do most of them depend on a lot of progress in robotics in the next few years?

13:52 Or if we were to stipulate that robotics is going to be slow, would that make you much less concerned about how the next five or ten years go? Or does rapid progress in just AI models get you to a lot of the bad outcomes on its own?

14:05 Yeah, I think I'd be less concerned, but not by that much. So almost half of the US economy is in jobs that are remote capable in terms of like GDP. And if you could fully automate the process of doing science and technology R&D, which would require some robotics, but like maybe not full spectrum humanoid robots, but more like automated wet labs and automated other types of labs combined with like super smart digital AI robots.

14:32 I think you could just see more and more the types of activities that create a lot of economic value, a lot of military value, and a lot of danger being fully automated. And there's also kind of crazy stuff that's enabled by... Taking that seriously means the AI would be as good at piloting robots and driving cars as humans are.

14:58 And you could have a scenario where humans wear augmented reality glasses and then are directed what to do in the physical world by these AGI systems. And you're seeing early versions of this and package picking, which has been happening for years. But that's one way where you could see a very fast change in the physical jobs as well.

15:19 Where it's not like humans go away, but they become de-skilled basically. Or somebody with no plumbing experience could put on these glasses and then compete with plumbers who actually know how to do it. And then I think super smart AIs would be integrated into lots of critical systems. They would have lots of opportunities to take actions in the real world because we'll connect them to our credit cards.

15:44 We will give them the ability to do lots of things on the internet. There's no way we're going to be tracking what they're doing all the time because we already aren't. And they'll be doing way more stuff in the future. And they could pay, coerce, persuade, trick people into doing things in the world.

16:03 And then they might just wait for the robotics to get there if they're super strategic and scheming. And to me, it's like... The bigger point is, That whether AGI plus human competitive robots are coming in five years or 20 years matters way less than whether it's coming at all. In the three-body problem, they know the aliens are coming in hundreds of years and they mount a full Earth-wide effort to prepare.

16:41 We're debating whether the aliens are coming this decade or next decade or the one after that. Mostly acting like business as usual. The amount of effort and money going towards preparing for AI in some large sense is just minuscule compared to so many other things that we're doing.

17:01 That makes sense. I feel like the speed of the transition could matter for a couple of reasons. One is that the disruptiveness could matter a lot. If you think about a professional like truck driving, it really matters if presumably 50 years from now there's not going to be But then the other thing is I feel like the...

17:31 I think there's a question about will the AI work for us or will we work for the AI? And if they're developing relatively slowly, it seems like that may leave more room to develop kind of a good political economy where humans figure out how to stay in control because the first five or 10 years, the eyes are maybe slightly smarter than people rather than like dramatically.

17:53 I think that the speed of the transition or

18:06 These different technologies coming online does matter. But one, I would say like gradual disempowerment of humans as a result of like truly universal labor replacing machines, both in the software world and then the physical world coming online, just feels pretty hard to avoid. And you could say like, well, humans will do relational work and maybe we could have the whole economy revolve around that.

18:30 The way like 90 plus percent of humans did agriculture for most of history. And then now it's like 2% in rich countries. And it seemed like crazy that that would ever work. But I mean, it was really rough going through that transition, but it did happen. I do think that we haven't seen that much job displacement from AI yet.

18:53 But I think the more important point is that it could happen very fast. Because when we look at charts of AI progress on capabilities, you see like these smooth exponential curves going up. But then when we experience AI progress in the real world, it's like these like tipping points. Where the AI is not useful as an agent and then it suddenly is useful enough to be worth using.

19:15 And the AI is below human level at hacking and now it's superhuman at finding and exploiting vulnerabilities in software. And so I think we could be in a world where we don't quite have AGI. It's not reliable enough. It's not able to handle long-term tasks or it's not multimodal enough or whatever the thing is.

19:34 And then you solve that last mile problem and maybe this happens soon. During recursive self-improvement where you could go from like, you know, systems kind of like what we have now and then maybe like a number of turns of the crank internally at the companies. And then you have like these broadly superhuman systems that would be like really hard to manage.

19:54 And there's questions of like if there actually is a super intelligence that's created, like. Is it going to just be accessible to anybody who wants to log in and pay $100 a month for it? Or is it going to be kind of closed off and used by governments to develop amazing technology and weapons and all kinds of stuff like that?

20:15 One of the reasons the pace matters, right? Because one of the patterns we see is that you have a frontier model that's a year or two ahead of the rest of the industry, but then over time people figure out, reverse engineer, figure out the tricks or the distill, and you end up with open-way models. And so a slow takeoff might be a world where everybody has a roughly similar supercomputer, superintelligence in their pocket, and that maybe leads to a more stable world where there's a fast takeoff where, you know, OpenAI or Anthopic has the superintelligence that's smarter than everybody else.

20:41 That might be a different world where people are much more vulnerable, right?

20:44 Yeah, I think it's harder, like the faster the takeoff is. But I mean, if we think about the world where like everyone just suddenly has a super intelligence, well, everybody wouldn't have it, right? Like the amount of it they would have would be mediated by how much money they have and how useful it would be for them would be mediated by their skill in using it, but also like their existing power.

21:04 The President of the United States would be able to get a lot more from it than us. I think this scary removal of limits on human power projection is the main meta thing I'm worried about. Where, you know, no human has taken over the world, but we've also never invented a new type of, you know, human replacer that you can scale up by orders of magnitude in a short amount of time.

21:34 And that could vastly exceed us. But even if it doesn't, even if it's just human level, but like superhuman at speed and coordination and ability to scale, then you could have a situation where it's like there's a AGI's and then a few billion and then a few trillion. And they're just doing the bulk of the economically valuable work.

21:55 And whoever controls them is going to be owning and controlling more of the future at the expense of basically everybody else who's going to be cut out meaningfully from that.

22:06 So what do you think of what's been happening in the software industry, where I think you could argue the cloud code today is, certainly in some respects, in terms of speed, is superhuman, certainly better than entry-level programmers. And there doesn't seem to, even there, which I think you would expect, if you didn't know anything else, you would expect there to be the place where you'd see disappointment effects.

22:24 There's like disagreement. There seems to be maybe some, there may be some disemployment for like the youngest employees, although that's maybe confounded by the fact that news rates went up a couple of years ago. But are you surprised that we haven't seen larger scale layoffs in the software industry as the, you know, as we've been developing coding agents that are approaching, if not exceeding human level coding abilities?

22:49 Yeah, I think a bit surprised because it does seem like coding by hand is fully optional and has been for a number of months now. And like software engineers I know who were not people who used AI ever just now don't code at all. It is like there are still humans making design decisions and they're kind of like moving up the layers of abstraction.

23:09 Yeah. I don't feel close enough to know how much, clearly Claude Code and Codex and whatever, they've been able to move up those layers and make more and more of the decisions themselves. But there's still people deciding what it is they're broadly trying to do. I mean, in the jobs chapter, I mostly use this canaries in the coal mine paper from these Stanford economists, which looks at relative employment and bisects by more fine-grained age categories and found that For the 22 to 25-year-old, most AI-exposed jobs, there was a nearly 20% decrease in relative employment, even after controlling for things like interest rates.

23:50 There's been some critiques of the methodology, and then there's responses from them, and they've changed the approach. I do kind of buy that it's harder to get a job as an entry-level software engineer because of AI coding agents. The people, yeah, there are more senior hires happening, and they're paid more because they're more productive.

24:08 But yeah, I think in a world where like you can actually have an AI system that's not just writing the code, but like making the decisions about what products to build and like what strategy to pursue and like what the company does in the first place, it's like a pretty different world. And so if you can have like this fully autonomous company where maybe there's a human that spins it up and it could be like the human has the kernel of the idea and then the AI just runs with it.

24:35 Spin up a new company that, like, makes a bunch of money. And then, you know, I'm going to own it because, like, I took the risk and invested in this thing. If you truly buy the AGI idea, it should be good at doing that. And you'll just see these, like, autonomous companies, like, doing things. And, like, maybe there are humans that are, you know, the legally liable person for something going wrong.

24:58 The, like, accountability sink. Like, I think we get into some questions about, like, You know, personhood and ownership. And like, I think it would be very dangerous to give AI's personhood as some people have argued. But yeah, I just, I think like my answer for all of this is just like, we don't have AGI, right?

25:16 We don't have a truly universal labor replacing machine. And so any claims about what will happen to the labor market, we just like haven't actually seen that yet with a caveat that like, yeah, I think it is a bit surprising how like, I guess one other point is you don't see layoffs that often because of AI.

25:35 It's more like you just don't hire people in the first place. But then if there's truly AI-native startups that are able to compete with these companies, then maybe they just will kind of force layoffs in the future. Yeah. But I think humans are not economically rational agents that are just optimizing everything all the time, thankfully.

25:58 And so I think it'll be slow until you get to these bigger tipping points we haven't seen yet.

26:07 Yeah, I definitely agree

26:08 with that.

26:08 I think you saw this with the internet as well, that you had... All through the 90s and 2000s, you had companies adopting new computer technologies and not doing a lot of layoffs. And then when 2008 hit, companies suddenly felt very much more pressed to squeeze as much value as they cut out of their workers.

26:27 And so they laid off a bunch of companies and kind of took advantage of

26:29 the increased efficiencies in ways that weren't

26:31 obviously great for the workers that were there. And I can imagine something similar happening where there's not a lot of layoffs until there's some future recession or other kind of crisis. And then suddenly the companies realized, oh, actually, we can get Yeah. Yeah. Yeah. To cloud code to implement.

27:05 I guess that's kind of what I expect for the economy as a whole is that there's a lot of, I mean, a lot of what software companies do is like interacting with users or customers, interacting with investors, figuring out kind of the logistics and things and like the kind of real world aspects. I think part of the optimistic view is that many parts of the economy will have that same kind of character where the job of the people at any company will be to interact with the other people and the people will prefer to have other people for various reasons.

27:39 Anyway, and then actually this, rather than making humans obsolete, it'll become a complimentary thing where the kind of technical details get handled by the AI, but the values questions and the big picture strategic questions are handled by the people because all the people are the stakeholders. They're the ones that benefit or are harmed by whatever decisions are.

27:56 Why does that not seem likely to you?

27:59 It's a good question. I mean, yeah, it'd be kind of interesting if like all of the kind of back office roles were just automated and then anything client facing, like you just have like these like human... It's kind of like the Shoggoth or something, but instead of like a smiley face, it's like an actual human.

28:14 And then there's like the Shoggoth, like, you know, doing all the accounting and all the production and whatever else. I guess if you're in a world where you have actual AGI, like fully multimodal, you know, you can't tell the difference, people on video calls doing this stuff. Like there will be demand for concierge, human, actual humans, you know, verified actual humans that you can talk to and meet in the real world.

28:41 But I think the incredible efficiency gains and cost gains from like fully automating as much as possible are going to create these like Really hard to compete with companies where I don't know, like, you know, Lemonade or something like is like one of these relatively new insurance companies that has like, it's so much cheaper than other ones because it's like super automated and kind of bare bones.

29:05 Yeah. I think for anything but the most luxury kind of markets, you're going to just see a lot of pressure on this to minimize the amount of human involvement. And it's still just like, what are all the accountants doing? What are all the people that were in those roles doing now? I think... I believe that the companies are trying to build machines that can just thoroughly outclass us across the board.

29:40 And they think they'll do it. And maybe there's some questions of, can it really write a great novel or something? Because there's intention behind great art. And bracketing stuff like that, The relevant question for the risk and kind of the effects on the world is like more like, can it automate its own R&D and can it automate all R&D

29:58 after that? Because I do expect we'll have shifts in the like relative distribution of different kinds of work. So I do think some category, you know, accountants that probably... Hardest to know for sure, but probably there'll be fewer accountants that you'll have a smaller number of accountants overseeing agents that do most of the actual accounting work.

30:14 But you'll have more people in the kind of care professions. And depending on how quickly robots go in the building trades and warehousing and things, at least until robots take those over. And again, this is why I think the speed of the transition matters, because if you're telling, you know, a banker that, oh, they have to go work, be a home health aide, like that's not going to go over well.

30:33 But if it's a 20, 30 year process and like new college grads, I mean, I think there's something you've seen is that the new, like hiring is very weak for underlevel programmers, but it's very strong for, I think it was like home health aides. And obviously it's not exactly the same people, but the distribution shifts where the, you know, because like doctors are also potentially, like my

30:50 wife delivers babies for a living. I don't think that's going to be automated anytime soon.

30:54 So anyway, I think the optimistic view is that over time, people shift, the workforce shifts to more jobs where there's part of the job is being a human being and like having face-to-face direct action with people and less to information jobs, which is probably bad for people like us. Like I like sitting at my computer typing things and if that becomes obsolete, I won't be thrilled about that.

31:13 But I think most people are not as attached to that and would maybe have perfectly happy careers if they're doing kind of more hands-on types of work.

31:22 Yeah, there will be A lot of people working in relational jobs and even the kind of dystopian futures I'm worried about up until like, you know, bracketing stuff like we are taken over by AI and like, you know, enslaved in some way, which I don't put that high probability on. I think I have way more probability on extinction than the kind of human factory farm scenario.

31:47 But yeah, I mean, I think... It still would just broadly devalue labor. And I think that will just in our current political economic arrangement be quite bad. If you have some kind of institution... In a world where you want to be bringing AGI into it, you have like really strong institutions that are fit for purpose.

32:12 And maybe in like the post-war era when we were like building a bunch of institutions in the wake of, you know, this calamitous destructive event, you could be thinking about like, okay, what do we do to prepare for this new thing? But right now we're in like this period of institutional decay after decades of kind of like Decline and you really just don't want to like throw this new.

32:40 Upending thing into the mix.

32:42 So I thought one of the most interesting sentences of the book, I was struck by reading through the book, that there are little tidbits of what seemed to me as skeptical views of this kind of fast takeoff view. And one of them is, you write that our world is already being steered by optimizing super intelligent entities.

32:57 They're called corporations. Many of the features that make them powerful, hard to control, and socially destructive are also present in AI systems with an entire ceiling. And I feel like that's something... Anyway, that's a point that I think I've made occasionally, that this, in some sense, the superintelligence may not be as alien as we think.

33:15 I mean, I think you're exactly right that in some sense, Google as a corporation can accomplish much more impressive things than any individual person can do. And that it does have some of these alignments. I mean, the leftist project is kind of pointing out all the negative aspects of having these...

33:35 A moral, profit-seeking organizations that are trying to maximize profit rather than accomplish other goals that we might prefer them to have. But it also seems like it's not the worst thing in the world. I mean, we in the United States do have a pretty high standard of living. And I think the higher ceiling is really the question.

33:56 One way I could see it going is if AI... Yeah, I think. Creating these like.

34:30 Powerful institutions that have these optimizing goals and then letting them loose in a poorly regulated world. I don't know. I think it's created a lot of predictable harm. I give the example of how meta... Knew that some significant fraction of their ads were for scams. And it was on the order of $50 billion a year from Americans going to these scams.

34:54 And Meta was making $7 billion a year from it or something like that. And the fine they were going to experience was $1 billion. So they're like, whatever, we're going to let the scams happen. And they had an anti-scam task force and Zuckerberg disbanded it. And so that's pretty bad. Obviously, that's not world-ending.

35:12 But I think that same kind of optimizing pressure combined with the removal of the natural limits that exist on humans, which corporations are ultimately composed of, is what makes this all very scary to me. We're in a world now already where swarms of AIs can find novel ideas. Zero-day vulnerabilities that all humans and past AI models had missed and exploit them in software.

35:36 And they can solve open math problems that have stumped people for decades or centuries. And this is still just so much less crazy than what the industry is trying I think a corporation composed of AI that can act at superhuman speed all the time, coordinate superhumanly well to achieve specific goals or profit maximization will find all kinds of ways to exploit loopholes in the economic and legal system.

36:13 And even setting aside the kind of I think Dean Ball has made this point where it's like, if you had AI looking through all of the activities that people were doing and trying to rigorously apply the law, people would be getting prosecuted left and right for all of these things. All the various things that they're doing that we like just accept as part of life because we understand that the law is imperfect and it won't be enforced uniformly and yada.

36:40 But like in a world of very good surveillance and like machines that can interpret the law, which I think we already basically have, this just breaks everything. And so many like the slot problem, you know, is like. Kind of trite, but kind of not. It's like the internet is just polluted with AI content that a lot of people hate.

36:59 A lot of people love is the other problem. And it's really hard to solve. And I think as these companies make the AI capabilities better and deploy them in more places and make them more autonomous, you're just going to see them finding all these weaknesses in our political, economic, social structure and breaking fundamental things.

37:19 Education is this thing. Younger people being taught by older people how to be adults. A thing we've done since time immemorial has been broken by these systems. And maybe we'll build something better from it, but right now it's broken. And we're going to keep breaking. I mean, I say we. These companies are going to keep breaking things and then expecting the rest of us to figure out how to fix it in a political system that doesn't know how to solve problems anymore.

37:47 I agree with most of that. I think where I am right now is I feel like There definitely are going to be harms from rapidly improving AI. And I would not be that sad if the process slowed down because I think there is going to be enough instability and that it would be good to have a little more time to adapt.

38:09 But I guess the question is like... One approach to that is, okay, there's going to be a series of challenges, and we decide we'll tackle those challenges and figuring them out one by one. So the one that I didn't think about most recently with hacking is when I started writing about this in 2023, it was kind of a theoretical thing.

38:24 Allens were not good at hacking, and I wrote back in 2023, this is going to be a problem in the future. We should be getting ready. People mostly didn't. It's a little hard to get people to Yeah. And it's been interesting, actually. When Mythos was announced in April, I tried to kind of ask the round of people in the security world I knew about, like, are people panicking?

39:03 Is there, like, a scramble to fix stuff? It was like, how did no? And the hugging face was a good story, but it wasn't like nobody died. The economic cost was not that high. And so it seems like we actually might have a year or two to prepare. Anyway, so I feel like the optimistic view is that, yes, AI will stress test a lot of our institutions and we'll have to fix them and there'll be some problems, but maybe we'll just be able to figure those out one by one and get to a new equilibrium.

39:31 Whereas the opposite view is that it's like an exponential, that the pace will accelerate a lot faster and for a while we'll be able to kind of But then there'll be so many changes that we can't figure it out. And I guess that has implications for what the solutions are. Because if you view it as just a series of things, like the education, for example, I feel like there's things like we can have people have written oral tests rather than working on computers, I think would solve some of these problems.

40:01 But if you view those as precursors to a much bigger thing that's going to happen in the future, then the policy solutions are, oh, we need to stop this. Whereas if your problem is mainly just it's going to put stress on our institutions that we'll have to adapt to, that maybe we can just adapt. So talk to me about how you think about what should be done in many policy-related books.

40:18 The back part of the book is about how to respond to this. How do you think society should respond to this challenge?

40:27 Yeah, I mean, we should stop this. The race to replace us. We should stop these companies from building AGI. I think we're going to just see these new problems come online before we figure out the old ones. There's this common argument that people make about how existential risk distracts us from all the near-term harms from AI systems.

40:50 And I kind of want to like uno reverse it where it's like it's distracting to be dealing with the implications of continued AI progress. Because as we're trying to figure out education and hacking and all these other problems, these new things are like threatening us in the future. And so to take that distraction off the table, we should freeze frontier development now.

41:11 And then I think we'd have a very, very large task of managing and governing the AI that we have today. And I think this is both because it's risky to build these systems, but also

41:24 because it's undemocratic.

41:26 The AI safety community has historically focused on the risk part of this and on aligning these systems with human intent, but spent very little time comparatively asking, do we actually want to be replaced by machines? I think the answer is obviously no for almost everybody. And my standard is that we should not be resuming developments of AGI or these universal labor replacing machines without strong public buy-in and a scientific consensus that the work can be done safely.

41:57 And what exactly that looks like, that'll be on the people who want to build it. I want to flip the onus from us having to justify why we stop to them having to justify why we continue. Yeah, we can get into how that would work or why that might be necessary if you want.

42:17 Yeah, I'm interested in that. So in the book, you say we should have a treaty between the US and China to agree not to develop AI systems designed to automate labor, which seems... A little vague to me. Fully automate labor. Right. Fully is doing a lot of work, right? Because like GPT-4 automated some kinds of labor.

42:37 Sure. And we have this jaggedness where over time, new versions of LMs automate more labor than previous version. When you say fully, what do you mean by fully?

42:45 I mean, I think just freezing development of the most capable and autonomous AI systems. So like Opus 5.5 and GPT-6, Gemini 4. These are the last models US companies get to make. And then they can productize. They can build more scaffolding around them. They can serve customers. But that's it. And there's like a tricky question of like, you know, China is behind the frontier.

43:13 And so like they'll want to be able to catch up and then stop there. And like that's a negotiation that would have to happen. But I think it's like, yeah, no new training runs, you know, models larger or as large as the ones that came before. No more reinforcement learning from verifiable rewards. Yeah.

43:35 And I think these companies know what they're doing that might advance the frontier and which is just like serving customers or doing product work. And so I'm not so worried about like the definitional part. I think there's like some subtleties and figuring out the edges of this is maybe a little tricky, but like you could just be overly inclusive and then dial it back from there because I think that the stakes of getting this right are incredibly high.

43:59 And there's this asymmetry where it's like, oh no, you like You accidentally foreclosed on some economically useful work that wasn't actually risky versus like, oh, you accidentally let them build AGI and we lost control of it and it was really catastrophic. And so, yeah, I think the US could ban this domestically quite easily.

44:22 You could put auditors in all the companies that have employee level access to the Slack and email in the offices. You could have more auditors than employees. You could overly define it. You could criminalize developing AGI. And the Bernie and Greg Kassar bill to ban superintelligence criminalizes pursuing it with up to 20 years in prison the way we do for trying to develop a nuke or buy a weapon.

44:45 And I think it really should be seen as this profoundly...

44:53 So I feel like the definition really is going to matter here because I think right now you can say, okay, there's two or three or four companies that are on the frontier. And if you get them to stop, that is going to delay progress towards AGI for a couple of years. But a big open question is like, Does AGI need to be at the scale of the current models?

45:15 Something I assume will happen is that over time, people will figure out how to get current frontier level models at lower size. We'll have smaller models, more efficient training methods. And you'll also have, you know, right now there's many, many companies training various kinds of smaller models for various kinds of niche purposes.

45:34 And so if you had a rule that said, you know, no models above this size, you have lots and lots of models just under the size threshold. People will come up with more and more ways to tweak it. And so you're going to have, I assume you need some other things. Maybe there's some set of benchmarks you can't exceed.

45:49 But there's going to be a lot of, if there's something you want to be a durable thing where 5, 10, 20 years from now, people are still not making AI. You'd be a pretty robust definition of what it is. Otherwise, you're just going to have the kind of lawyers will say, okay, how do we stay just under this technical limit while still building something more powerful?

46:08 Yeah. How do you think about that?

46:10 Yeah, I mean, I think that's all fair. I think that the argument I make in the book is like, if you give politicians enough of a what and a why, they'll figure out the how. And you and I are not going to figure out like the exact definition here. And it wouldn't matter if we did. Politicians need to feel the pressure to actually do this.

46:31 And if it became like the, you know, nuclear agreements in the Cold War, like there are lots of questions that were similar to this about how to define these agreements. And there had to just be a lot of pressure from like the Cuban Missile Crisis and like near misses and all these things that happened.

46:47 But like nuclear weapons are a pretty clear cut thing, right? If you build a thing and test it, it either blows up or it doesn't. And there's not, you know, there's not like You haven't seen that on the dark web?

46:57 Yeah.

47:18 Another analogy that comes to mind here is like when OpenAI signed its deal with Microsoft, they have this clause that says when we reach AGI, then we will X, Y, and Z things will happen. And I think plausibly anytime in the last couple of years, OpenAI could say that, oh, you know, Anyway, any of the recent models could be plausibly described as AGI.

47:41 But that was just understood as this kind of nuclear option where it was in everybody's interest to kind of accept the fiction that this is not AGI. And I feel like a similar thing might happen where it's like, okay, we all agree we want to quote unquote super, we won't build quote unquote super intelligence, but then we'll release this model that's like a little short of super intelligence.

47:59 And it's And, you know, nobody wants to have their friends thrown in jail for 20 years. And so it could just become this, like, theoretical power the government is next. I guess I just feel like it's really important if you're going to have a bright line, it needs to be a bright line. It can't be this line where there's, like, lots of things on either side and people can disagree about, you know, which side of the line any given system falls on.

48:19 Yeah. I mean, I think you could just set like you can't train a model bigger than like some size that's maybe like quite a bit smaller than the, you know, than GPT-4 even. And just like Ezra Klein talked about banning recursive self-improvement and like one very like, you know, It's like just don't let them use coding agents on AI R&D.

48:42 That seems even crazier to me. Like you're going to say like everybody in the world is allowed to use a coding agent to accept OpenAI and Anthropic or like a list of...

48:51 It's like, you know, if you're looking at somebody in like a wet lab that's dealing with like, you know, pandemic pathogens and then it's like, oh, it's crazy. You don't let them like go in in a t-shirt. It's like, yeah, no. They have to go in the really annoying seat.

49:03 Right now there's like three or four companies that are the only ones close to AGI or... Or superintelligence or whatever. But if you stop development, lots of other companies are going to gradually improve. And you'd have to have some way of adding new companies. If any company gets too good at that, we're going to strip them of their cloud code permissions.

49:23 That seems like a hard equilibrium to maintain over more than a couple of years.

49:28 I think you could just be tracking which companies have which amount of compute. And then any company with this much compute is just there now in this world. And maybe you'd have this thing where... Or maybe it's like a sliding scale, like a gradual kind of thing.

49:41 Because I feel like I was at the curb a couple of weeks ago and I talked to David Kruger, who he was pitching on me his plan. I mean, the conversation was off the record, but he said the same thing publicly. I think that he wants to basically ban production of new GPUs and the only... You can build hardware that's designed to only do inference, not do training.

50:01 But basically, make it flatly legal.

50:03 You can cut the interconnect on the chips and then they would not be able to do training, but would be able to do inference.

50:07 Yeah. And I guess I feel like if I believe this, I would... I think you'd want something much of a draconian, not like a law, but basically stop producing new hardware, which is not something I favor, but I feel like maybe that's the position I would push you to if I pushed you really hard on this. Because if you think it seriously enough, I think that might be the only way to actually prevent continued progress in general AI.

50:31 Yeah. This is a point people often make where they're like, well, if you pause AI, there'd be this compute overhang because the computing power would keep going up and then it'd just be too easy to build a really capable model and it would happen. This iteration would go from maybe one generation to five generations of progress.

50:48 And then I'm like, well, you're not taking the hypothetical all the way. If we're in a world where there's enough political will to pause frontier development, You could also push for pausing AI chip development or stopping making new chips. I'm a radical out here saying we should stop, and David's got an even more radical position on how much.

51:10 Yeah. Even if that's a completely reasonable thing to do and a morally and intellectually serious thing to do given the stakes of what's happening, which I think is true, I also want to be a little bit mindful of, okay, can we get most of that without completely nuking the business of this giant company that's already going to be fighting this tooth and nail?

51:49 Maybe you just bite the bullet and say, yeah, no, NVIDIA doesn't get to make AI chips anymore or they can only make inference chips. Inference chips, there's going to be a lot of demand for inference of the models today. And in a world where they're not able to train new ones, there's going to just be like more compute available for doing, yeah, serving customers.

52:08 But yeah. And then the like algorithmic efficiency point, I was talking to somebody at the curve and, you know, very competent alignment researcher. And like they agree with me that we should stop, but they're quite worried about the algorithmic efficiency and like basically I don't want to say this is an easily solved problem because we could stop what's happening today overnight because you have to figure out a sustainable regime for this.

52:31 But I don't know. Mirror bacteria is this hypothetical thing which is like... Reversing the chirality of molecules and if you make it at all it could like just kill everything because we have no defenses against it. And the scientists were looking into this were Like we can't rule out the total destruction of the biosphere, so we should just not pursue this.

53:06 And they got a global moratorium on further research into it. And it's disanalogous in that like there weren't that many people working on it. The risks were like extremely credible and the benefits were like not as obvious. I think there are potential benefits, but like it's not like AGI where it's like, well, yeah, you can see how that could be very useful.

53:29 And yeah, but like, could somebody be working on that in secret? Like, sure. I think it'd be quite hard to like actually surveil everybody who has a relevant experience without creating all these like civil liberties risks and whatever. But I think... Creating a stigma against it. In my vision of the future where we stop this in a sustained way, it's not just like a bunch of experts or elites got together and we're like, all right, we have to stop doing this.

53:59 It's like the world understands that this is an incredibly dangerous thing to do. And the international community reacts to any kind of discovery of a project that's trying to do this. Like, Right. Right. Right. I think it's tricky because somebody with a computer writing code and trying to make a transformer more efficient or coming up with some new architecture could be making meaningful progress on the cusp of a theoretical breakthrough that could make AGI possible with compute less than the amount that we might set the bar at.

54:54 And that's a hard problem to solve. But I think in a world where it's illegal to do this, corporations are just going to not risk doing it. There's lots of stuff that companies could do and maybe they could get away with it, but they don't want to risk going to jail or their corporate charter or whatever it might be.

55:13 And so I think it's going to take so much political will to actually pull this off that we're in a radically different world than the one

55:21 we're in now. That's it for this episode of the AI Summer podcast, which is edited by Vulgate Media. You can find more of my work at my newsletter at understandingai.org. I'll be back soon with more voices from the AI Summer.

55:35 [music]