Stopping The AI Safety Cult ft. Adam Becker & Cal Newport

Call zone media.

My name's Ed Citron, and this is Better Offline.

Better Offline.

I am back, and I'm announcing that we will be speeding up the process of researching powerful and dangerous podcasting technology. My researchers are spending over $ 200 million a day on GPU compute, trying to create and generate a recursive self-improving Zitron, and no amount of pressure from world governments will stop me.

But before my greatest work is complete, I must continue to record. And today we've got an absolute barn burner. I'm joined by the claviculars of computer science, by which I mean Professor Cal Newport, an astrophysicist and author of more everything forever.

Adam Becker, fellas, welcome back to the show.

Thanks for having us.

Yeah, I'm terrified of the idea of recursively self-improved. That means your newsletters will be 200,000 words long now? Is that what we'll expect?

Yeah, they'll either be 200,000 words or 100 words long.

Oh, maybe it's not the worst thing then. All right, keep going.

When the perfect matrix of Zitron is complete. Well, today though, this was inspired by your wonderful piece, Cal, in the New York Times about the rationalists. And we are going to start very simple. Who are these people and why are they suddenly notable?

Well, I'm probably going to actually throw the baton to Adam here in a second to give the history of this group because He and his book was one of my primary sources in trying to understand them. But let me just set the stage for why I was writing about them. I don't think people realize just how weird some of the lead figures are who are at, in particular, OpenAI, Anthropic, and DeepMind. And I don't think people realized that until last week when you had a whole group of 20-somethings casually debating in their lunch hour whether it's 10% or 50% chance that we're all going to die within the next couple of years. And that, I think, caught a lot of people's attention. I'm like, wait a second. What is

going on over there in those labs? Why is this type of rhetoric only coming out of those particular labs? Why does Jinson Wong think that this is ridiculous? Why is China laughing at this narrative? There's something going on here that's not right. I think people are starting to wake up to this. And I think it really is this family of futurist philosophies that come out of the Bay Area that have mutated and split in various ways that are really defining our current moment And no one really realizes it. But Adam spent years, God help him, immersed in this world, has read everything they've read. He knows the history well. So actually, Adam, I'll throw the baton to you. Maybe you can give us a story

of rationalism and effective altruism and where they came from and how they've split. To me, it's a fascinating, odd story.

Oh, God.

I mean. My standard response when people ask me this is, well, I could write a whole book about it, and I did. And I'm not going to get into the same level of detail I do in the book. But basically, this is something that was spawned out of transhumanist message boards and email lists in the 1990s.

And what is transhumanist?

Yeah, that's a good question. Transhumanists, I guess the shortest good definition I could give you is transhumanists are people who don't want to be human anymore. They want to use machines and especially computers to basically escape human bodies and solve the problem of death.

Okay.

Yeah.

So those guys. Yeah.

Normal, normal shit, normal shit, normal, normal way to do. Yeah, exactly. Yeah. And, um, And so, you know, these are people who are really excited about the prospect of AI, the prospect of, you know, nanotechnology that could, you know, basically serve as a kind of magic to do whatever they wanted. This was the 90s.

Nanotech was like the hot thing. And it was a bunch of ideas that basically come from science fiction. Right. You know, they read too much sci-fi. and confused it with reality and thought, oh, well, because all of the sci-fi I read takes place in the future and all of the sci-fi I read has super intelligent machines in it and nanotech and people uploading their brains into computers, that must be what's going to happen in the future if we just work hard and wish hard enough for it.

And so just so I'm clear, this is... This is a certainty for them. They're 100% sure this will happen.

They're 100% sure that they could make it happen, right?

Oh, so it's in their hands.

Yeah, yeah, yeah, yeah. Like, you know, they think, oh yeah, this is going to happen unless something, you know, ends civilization first or, you know, nobody listens to us or something like that. They think that they're working on the most important stuff in the history of humanity, basically. They think that they are going to uh, you know, make themselves immortal.

And when you say working on, that seems like a fairly broad thing though, because a lot of these rationalist types don't actually work on the stuff. They work next to it. Yeah. It's philosophy people. Yeah.

I mean, it's not even so much that they're working next to it. It's like, they talk about it. They like to talk about the prospect of nanotechnology. They like to talk about you know, super intelligent AI. They like to talk about these things. But yeah, so these people spent a lot of time doing that on message boards in the 90s. And there was this one guy who showed up in the mid to late 90s on one of the most sort of influential of these message boards, the Extropians, which, you know, don't get me started on that name. It comes out of a misunderstanding of physics.

Yeah, that's an annoying name.

It sure is. That's how you know this is a serious group of scientists.

That's right.

The extropians. Yeah, that's that's correct. Yeah, that's exactly right. And they're like, yeah, we're going to fight the second law of thermodynamics, to which my response as a physicist is, you know, yeah, good luck with that. But basically, this guy, this kid shows up named Eliezer Yudkowsky, who is a teenager at that point.

Dropped out after middle school, dropped out of school. This is like when he dropped out.

That is wild.

Yep. Dropped out after eighth grade. That's correct.

That's crazy, man.

Yep. And yeah, Yudkowsky then goes on to write an enormous amount of his own deep thoughts on what he thinks the future of AI is going to be. and relatively quickly comes to the conclusion that AI is going to, you know, he takes an article of faith among these people, which is that AI is going to become self-improving and lead to a super intelligent AI at some point in the relatively near future and lead to a singularity where everything changes and, you know, all of the stuff that the transhumanists believe is inevitable is going to happen. Yudkowsky then comes to the conclusion that such an AI might not be friendly to humanity and probably wouldn't be unless it's done properly.

And the singularity is when the computer is smarter than us, just so that I understand.

It's really ill-defined. It's when the computer is smarter than us, or maybe it's when the computer can upload us into more computers so we can all be computer and everything is computer. The singularity is when a lot of tech, but all the time and everything. It's not a well-defined thing.

Cool.

Yeah.

And this is the thing they obsess over the thing that's ill-defined. Oh yeah.

Yeah.

They, they think that that is pretty much inevitable unless, you know, we have like a nuclear apocalypse first or something.

This is worth, let's just pause on this for a second.

Right.

I think it's really important because these ideas that I was talking about, so I'm very familiar to what people are talking about today. They were talking about this. This is about the early two thousands is roughly when you Kelsey kind of had this change of heart of, um, The superintelligence, of course, is inevitable, but now he was worried about it instead of being excited about it.

This was all thought experiment. Okay.

I mean, there's no LLMs there. This was during an AI winter from the computer science perspective. It was entirely inductive thinking directed by ideas from sci-fi. So they just sat there and thought about this, right? So this is not an engineer looking at a particular technical trajectory and saying, uh-oh, I'm worried where this is going to go. This is people just thinking through the thought experiments of, well, if AI could recursively self-improve, it would get super intelligent. What would it happen if you had a super intelligent AI?

And then just sort of exhaustively working out in their own minds various scenarios about, well, maybe it would be friendly, but maybe it would trick us. And what if it did this or that? And so I think that's important. So you have all these ideas are established well before there's any particular technology that people are worried about, established entirely based off of, at this point, thought experiments.

Yeah, I think that's exactly right.

So this isn't through proximity to actual scientists?

Nope.

So nothing about witnessing the actual technology following, other than arm's reach?

Yeah, no, I mean, it is exactly as Cal just said, and understandable. way of phrasing that is it's, it's essentially a kind of extremely dry fan fiction that they're engaging in. They are writing, you know, anxiety fueled scenarios based on these ideas that they got from science fiction, thinking that that's what the future is going to be.

Do they believe it?

Oh, yeah.

So they're sincere. They're just odd.

Yes.

Yeah.

And at the time you can like dismiss them as sincere oddballs. What happens is Yudkowsky just keeps writing and writing and gets invited to contribute to some group blogs and then spins off his own sort of blog called Less Wrong, which becomes a community hub for people. Yeah, that's him. But the other thing that happens is it's around this time, like the late 2000s, early 2010s, somewhere in there, that Yudkowsky meets Peter Thiel.

Oh.

And Peter Thiel gives Yudkowsky a bunch of money.

How old is he at the time?

Oh.

We did this math. We did this math, Adam, with the fact checkers.

Yeah.

This is like the first decade of the 2000s. He was like mid to upper 20s.

Yeah, something like that. Yeah, that's right.

Huh.

It's like a decade of him, a decade after he- Peter.

Thiel just giving a child money.

I mean, that's not out of character for Peter Thiel.

But he literally pays people not to go to college, right? Exactly.

The thing is, though, the Thiel fellowship where it's like, I'll pay you not to go to college to build something. This is just like, I liked your blog, kiddo.

Pretty much.

You think the world's going to explode. You better blog about that some more.

They met at, I'm trying to get this right, but they were at an event called the Singularity Summit.

I believe that's where they met. Yeah. Or maybe that's where, it gets complicated.

They definitely both attended.

It gets complicated. Yeah, it does.

Yeah.

But a lot of money enters the scene, right? This is where a lot of money starts entering this world.

Yes. Yeah.

Yeah, yeah, yeah, yeah. And Jadkowski- along with some other, you know, transhumanist, futurist types like Ray Kurzweil, starts this, you know, set of conferences, the Singularity Summit, and, you know, just gets people together every year to talk about this stuff. And the community of rationalists on Less Wrong just, you know, start, you know, gets bigger and bigger, and they start meeting up.

In the Bay Area, they start living together in group houses. What does rationalist mean? Rationalist basically means you spend a lot of your spare time reading Yudkowsky's writing. Yudkowsky thinks at this point that if you don't believe that the singularity is coming and that AI is an existential threat to humanity unless you figure out how to do it right, he thinks if you don't believe those things, you're not thinking rationally. And so he decides to post a lot about how to think rationally. That's it? Yes, that's it.

That's it? That's the entire rationality part? Yes. It's not about living rationally. It's just about their so-called rational belief that the computer might wake up. Are you fucking kidding?

Well, they apply it to their whole lives.

Yeah, exactly.

That's why the two big blogs are called Overcoming Bias and Less Wrong. It's all about this sort of like really sort of tech obsessive way of, I'm going to be super accurate in the way I think, right? And this is, I'm going to, I'm, I am not going to be overcome by cognitive biases. I'm not going to be tricked. And if we can just be like hyper Sherlock Holmes is like, that's how we're going to unlock the potential of the world. But Yukowski thinks like the ultimate application of this is preventing the extinction of humanity.

by preventing super intelligent AI. And it like, that is why we have to raise an army of like hyper rationalist people. It's our only chance to survive the, the coming AI apocalypse. So like that kind of becomes his obsession is that like this idea of like hyper rationalism, um, that he's the leader of is going to be, that's the John Connor, uh, this Skynet scenario.

Right. But they don't seem to be doing anything other than talking.

Yeah, no, at this point, they're not.

They're doing some things. I mean, they're in the polyamory. I mean, there's some.

Other things going on. But like, to be very, very clear, Yudkowsky is anointing himself as an expert on, you know, the art of rationality and cognitive biases and how to think. And, you know, this is an entirely self-trained, self-taught guy whose experience of the world is pretty limited.

And.

He is posting hundreds of thousands of words about this on last wrong and then starts writing as, uh, you know, a combination of artistic outlet and recruitment technique. Uh, he starts writing literal fan fiction, a Harry Potter fan fiction called, uh, Harry Potter and the methods of rationality.

Oh yeah. Yeah.

It's like 600,000 words long or something.

Yeah, it's longer than War and Peace.

I'm sorry, even I would never write something that long. Even I would... 200,000 words on anything is terrifying.

I mean, this is 600,000 words.

But what the fuck was it even about? Was it like.

Oh, it was about Harry, who is, you know, I think fairly clearly an author self-insert of Yarkovsky.

Right.

Harry being the specialist boy and finding ways to, you know, change and save the wizarding world of Harry Potter using his special logic and science powers.

Through logic and reason.

Yeah, exactly.

Yes. Right. Well, why? Great question. I guess it's just creative. Yeah.

I mean, I interviewed him for my book and that was what he claimed. And I, I believe that it started out as a creative thing, but you know, it was also very clearly him playing with these ideas that he has about, you know, this is the only way to think clearly. And, uh, and it did end up becoming, you know, quite a recruitment tool for the rationalists.

Um, And just to put a little, just make sure we have like the timeline beats clear and then Adam fact check me here. There's sort of this shift that happens where in the early 2000s, actually, I think his first money was, it was Peter Thiel, but it was also that other, that other internet entrepreneurship couple. They fund something called like the Institute for. ai singularity or the what.

Oh yeah the singularity institute for ai that was originally yeah originally the funding comes from this couple the name of which i don't remember but eventually like a few years after that in like the mid to late 2000s he meets teal and teal gives them like real.

Money yeah and this is where so as the 2000s move on so that in singularity institute was super intelligence is going to be a beneficial godhead that's going to redeem history well it's He shifts pretty. I mean, my understanding is like once that's a full-time job and he's thinking about it a lot, he shifts pretty quickly that he does that first big report. He shifts pretty quickly, but like, actually, wait, I don't think we'll control AI.

We should not build it at all. It's going to kill us. Like there was this shift that happened like kind of quickly in that area and he shifted valence and then eventually they renamed it to Mira Miri.

Yeah.

Yeah. That's correct.

Yeah.

So, so just, just to get the timeline straight here, um, Yudkowsky makes that shift from, you know, super intelligence is going to solve everything and we should be aiming for it as fast as possible to, oh my God, it's going to kill us all. He makes that shift around the same time Singularity Institute for AI is founded in the early 2000s.

Right. Is there an antecedent or does he just one day wake up?

I mean, he's just been on these message boards for a long time and he also saw that like you know, the nanotech guys who wanted a lot of the same things and made a lot of the same promises back in the 80s and 90s, people like Eric Drexler, they had their own sort of institute. And Yudkowsky, you know, at this point, he's, you know, unemployed and about 20.

And I think he's like, I want to work on this full time. Can I turn this into a job? And finds a way of doing that by you know, co-founding this Institute. But, um, I, I also, I want to, I want to just push back on one thing that you said, Cal, it's not quite that, you know, Yudkowsky wants to stop AI at all costs. He, he wants to stop AI at all costs until he and others figure out how to do it safely and At which point he wants to go full speed ahead. He still believes in the AI utopia if you can get a super intelligent AI to not kill everybody.

So he just thinks that that problem is harder than he originally imagined. So he's like, this might take us longer.

There's an impedance towards him being the thinking king. Like the thought king that can tell the computer how to be best, that loves us.

That's basically, yeah.

He will be the right-hand man to the digital god. But Adam, is this right? There is no technical antecedent for this shift in his thinking from actually this might be harder. The best I could tell from your book and look at other sources is that the shift in thinking just came from as soon as he got the funding for the Singularity Institute that he was helping to run, he had this big writing project he took on where he's like, okay, I'm going to produce this big report on superintelligence. And it was in just the internal you know, cognition of thinking through this report that he just convinced himself, oh, wait, I don't think we're going to be able to control this as easily as we think. Because I think it's

really important to emphasize, again, this is not someone who has seen something in a technical lab that is scaring him. All of these developments and his thinking at this stage, it's really just, it's Yukowsky in his own mind.

I think that that's a really important point. Is that consistent through time, though?

Yeah, I mean, from that point on, he's convinced of this.

But he's not seen anything, and these people have not seen anything.

No, I mean, you know, he later on, he starts pointing to various advances that are happening in machine learning as evidence that the amount of time that we have to solve this problem that he came up with uh, is, you know, running out. But no, you know, that original idea and his, his faith in it is based purely on thought experiments and conversations he has with other people in this community.

Because the way these fucking people frame themselves when they talk about this, they talk, I'm speaking from, I don't have a podcast episode I just did while I'm They don't talk about, and maybe this is how they varnish it, they don't talk about that they would be the authority in some cases. They're just like, we should have one.

That's how they always get you, these people. People like this. It's always, they sound reasonable, and then behind closed doors, trousers come off. Okay, that's not true, but, well, maybe. Yeah. But it's about how, oh, actually, we are like Boris Balkan in the Ninth Gate. I alone am worthy to deal with this. That's terrifying. I don't like that one bit.

I mean, oh, okay.

Yeah, because we're going to make this worse now.

Yeah, right. All right.

So this then becomes.

The key.

This is one of the key arguments in my New York Times op-ed. Like, why is this relevant to right now? Well, the issue is, so this world is going on. Yukowsky has his followers. He's convinced of this. It's mainly thought experiments, right? There's not a particular AI technology I'm looking at. That world severely overlaps with open AI, Anthropic, and DeepMind. And let me give you like the very condensed version, and Adam can add the pieces that I'm missing here, right?

But let's go back to the founding of DeepMind, right? There's three co-founders of DeepMind. Two of them, Hasabis and Shane Legg, they're at a Singularity Summit. They are talking with Yukowsky. Yukowsky says, let me introduce you to Peter Thiel. And Peter Thiel gives the initial funding for DeepMind. Shane Legg has been deeply involved in sort of the Yukowsky rationalist world for a long time. He was motivated by a talk that Yukowsky came to give about superintelligence at the place where Legg worked. He went on to write a doctoral dissertation on superintelligence, a frequent poster on Less Wrong.

So DeepMind has these deep connections to that community. When they get bought by Google— You know, that's when Sam Altman and Elon Musk are like, well, that's bad, right? Because this is, in a true rationalist sense, this is the technology that's going to either redeem humanity or kill us all. We can't have a company like Google control it.

That's when Musk puts up the money for OpenAI. You have Sam Altman, you know, tweeting favorably about Yukowsky at this point, saying he's done more for AGI than anyone else. Yukowsky should get the Nobel Prize. The current president of OpenAI, Greg Brockman, was running a less wrong reading group at the company where he worked before.

Wait, even Brockman?

Brockman was the normal one.

No, no, no.

All right, so you have these deep connections. The board of OpenAI, half of that board comes strongly out of the sort of rationalist EA communities. When they tried to oust Sam Altman in 2023, as Cade Metz reported at the time, it was the faction of the board with rationalist EA members. connections that felt like he was being insufficiently aligned with rationalist-type ideas. The temporary CEO that the board put in place to try to replace Sam Altman is someone who is so entrenched in rationalist circles that there's a character named after him in the Harry Potter fan fiction that Yukowsky wrote. Dario Amadei and his sister and the other employees who left OpenAI to start Anthropic did so because they felt like, again, they were being

insufficiently aligned in part to rationalist ideals about how do we usher in superintelligence safely. You have Amadei a couple of years ago parroting common rationalist talking points like, yeah, we have about 25% chance that AI is going to kill all of humanity. So these worlds, those three companies- They come out of this world of rationalism, which had a really big foothold in the Bay Area of the time, which means our leading AI labs, the way they think and talk about AI is heavily influenced by everything we just talked about.

Yep. No, that's all correct.

So when they say alignment, because alignment as an idea is not the worst idea. It's making sure the models give good outputs and the outputs aren't inherently evil.

But That's not really what they mean when they say alignment.

No, but that's exactly what I'm getting at, though, because there is the term alignment, which means the thing I just said to most people I talk to, and then what they mean by alignment, which appears to be something else.

Yeah, no, what they mean by alignment, and I mean, this is basically word for word what Yudkowsky says, is an AI that, when it becomes super intelligent, will not kill 100% of humanity.

Wow. Yeah. That's fucking annoying.

Yes.

I know I meant to sound more eloquent than that, but that makes me very frustrated.

Well, to make you more frustrated and to make it even more annoying and infuriating, because these guys were so deeply entrenched and involved with the AI industry, and because they'd been writing about it for so long, it means that when the sort of AI moment arrived for the public with the release of ChatGPT in late 2022, the rationalists and effective altruists were able to convincingly portray themselves as experts who had been working on AI for decades.

And this is how you end up with things like you know, Bernie having, you know, a sit down with Eliezer Yudkowsky.

Right. This makes more sense now because the way they frame themselves and they'll say like, oh, AI 2027 has been accurate in some ways even. And as you'll know, in a podcast coming up, I pushed them on that and they refused to actually open the blog in front of me, but it's, it's how they have scary scary stuff framed in the terms of academia without actually being academic right yeah.

Yeah cool now there's a two important sidebars here so sidebar one we've been kind of casually referencing effective altruism yeah this is important right because uh so effective altruism this is where it gets complicated so this is a another movement that has like similar sensibilities that became intertwined and heavily influenced. They sort of mutually influenced each other, right?

But you have effective altruism. It comes out of a bunch of utilitarian philosophers at Oxford, and they have this idea of, oh, we can apply. This should sound familiar, but it's like if we're very rational and empirical about altruism, we can do altruism better, which I think is like modern utilitarianism. That's a fine idea. This is where they're going around and doing research like what are the actual best charities to give money to? How do we measure that?

They have these ideas like, actually, if you're at Oxford, like an elite university, the best thing you could do for charity is to go get a high-paying banking job and then give up half your salary to charity. So they're trying to think through rationally, how do we do altruism as effectively as possible? Well, they overlap and are influenced by rationalism. And pretty quickly, you have a large segment of effective altruists who say, actually, the most altruistic thing we can do is prevent an AI apocalypse. Because why bother, why care about, you know, saving 20 million people now from malaria when we could instead be solving, you know, saving a hundred trillion future lives that will never be born if AI kills us all. So effective altruism,

not all of it, it's a complicated field, but big segments of effective altruism. And this is where Nick Bostrom comes out of at Oxford talking about existential risk and superintelligence. A big segment of effective altruism really begins to care about superintelligent AI as one of the core things to care about. And a huge amount of money pours into effective altruism. We're talking hundreds of millions of dollars starts going to these institutes and groups.

And nothing's been done yet, though. I just want to be clear. No actual actions have been taken to stop superintelligence. No. safety standards. No actual science has taken place either.

Well, they wrote the Harry Potter fanfare.

Okay.

So no, I'm not even being sarcastic. Like they, they wrote some posts.

They wrote some posts and, and kept talking about how it was really hard because, you know, what does it mean to align something with human values? And like, well, maybe your question is ill-posed.

Maybe go work on a fucking large language model. Go and work on.

AI instead of this weird thing. I mean, some of them did, right?

And that's how we got Anthropic. Yeah, yeah, yeah.

And DeepMind and OpenAI. But like, Just to everything Cal said is right, except one thing, and I think it's important. Nick Bostrom is definitely like part of the whole effective altruism community. But Nick Bostrom, he wrote, he's probably best known for a book that he wrote called Superintelligence that came out, I don't know, a little over 10 years ago that made a big splash, basically, you know, expounding on all of these ideas. And saying, yeah, you know, superintelligence is really scary and we should figure out, you know, how to control it or stop it before it kills us all. And again, this is before, you know, transformer models and the current AI moment. And it's not from any particular thing he's looking at. But the thing is, Bostrom,

who at that time is a philosophy professor at Oxford, came out of the same, you know, place, basically. as Yudkowsky. He was on those same 1990s listservs, and he's the one who sort of is most responsible for the connection between rationalism and effective altruism, because it was when his colleagues at Oxford were sort of working on effective altruism that he went over to them and said, oh, if you're worried about that, you really need to be worried about the AI apocalypse.

These are two different worlds.

Yeah, but at this point, they're kind of conjoined. Yeah, at this point, they're like, they're really like, you can't disentangle the two communities.

Yeah, there's a talk you can find. So William McCaskill was one of the two big utilitarians at Oxford that helped found effective altruism. And I'd crossed paths with him sort of earlier on in the EA world when they were really caring about, especially like the idea of donate most of your salary to charity. There's a talk you can find and I– it's on YouTube and I forget exactly when this happens. But there's a famous talk you can find in McCaskill where he basically comes out and says this is the new thing we should care about is existential– the existential threat of superintelligent AI. This is the sort of long-termism. So they used this– there's another idea that comes out of Silicon Valley. But this long-term

view of what's going to happen over the next thousand years is actually if we're just going to do the math. 100 trillion future lives matters more than, you know, 50 million lives now. So he kind of has this moment where he gives this big speech that where he's sort of announcing this is the new direction for EA. And so now they're like roughly interchangeable. This is why like if you read like Cade Metz's reporting on like the OpenAI board or reporting about Amadei and his philosophical connections, they'll sort of use the terms rationalism and EA interchangeably.

They'll use one or the other, say both. And that's why these circles have become very Very simple.

And they seem to have had different goals that have now kind of conjoined.

Yeah, I think that's correct.

Yeah.

I should also point out, you know, the most famous example of someone who took McCaskill's advice and, you know, went off to make a bunch of money in the finance industry and then tried to donate large amounts of it is Sam Bankman Freed.

Oh, yeah. Yeah. Big EA. Yeah. Yep. Yep. And actually, early investor in Anthropic.

Oh, yeah.

Yeah. Early investor in Anthropic gave a bunch of money to rationalist and effective altruist community organizations. And yeah, no, I mean, McCaskill met SBF when he was a college student at MIT.

Get around with a lot of kids yeah well they're the charlie kirk style thing where they go to college campuses almost yeah well i mean for thinking online dudes.

That's the thing they they do do this there's a uh god there's a piece i want to say it was in the washington post a few years ago by natasha tiku about uh campus recruitment strategies that the effective altruists were into because they have you know just enormous piles of tech money that they were using to you know, get them while they're young.

And we're talking like the effective altruist will come in and someone will just sign a check for like $ 60 million to start an Institute. Like we're talking huge, huge money coming from tech billionaires.

Yeah.

Because think about it. Who is the hero? Who are the heroes in the rationalist? You are Neo from the matrix. You're John Connor from the terminator. You are the people who are going to save the world. And if you're a, a tech, you know, a hundred million, a cent a millionaire, a billionaire, you're like, yeah, that tracks. That kind of matches my understanding of myself.

And I assume the EA rationalist types don't think, hey, wait, are we just playing into the hands of the richest people alive?

I mean, they kind of either don't think about it or don't really care. They don't think, they think that if you just think really, really rationally, than things like conflict of interests and- Oh, right. Corrupting your motives. Well, you know, those are just cognitive biases and you can overcome those by, you know, the power of pure logic.

That reminds me of this guy, Alex Heath, who is ostensibly a journalist, who has also become a venture capitalist this week. I saw someone comment on his post saying, now you've got skin in the game, you can be truly objective. And they were serious.

Oh my God.

But that is, that it sounds like the rationalist people. It's like, now you know the stakes- you will be bereft of any sorts of human biases you may have and work towards this vague goal we all have that none of us can describe.

I mean, they believe that, you know, making bets on things is a way of thinking clearly about stuff.

When you say making bets, what do you mean?

Money.

I mean, like, you know- Like what are they betting on?

Well, I mean, this is all they're like, you know, decision theory stuff, but they have, um, there's this, uh, um, forecast gambling website called, uh, uh, manifest, sorry, manifold, um, uh, where they, you know, like bet on, you know, it's, it's a prediction market. Um, and they, they love prediction markets.

Because economic theory, right? You're going to be the rational consumer. When your money is on the line, that's going to help you overcome biases and get towards all this theory behind it. You're going to get a clear, rational solution.

Yep.

So this reminds me of just the most annoying early people online.

Oh, yeah.

They were. People were doing mind-master shit, being like, yeah, we're so smart. Oh, well, do you want to make a gentleman's bet about who is the smartest? And then they said, Very strange, vague goals that they can move the goalposts on. Yeah, yeah, I've seen this before. Except they control so much money now.

Well, okay.

But now we've got to connect us to the current moment because it gives you– this is the argument I was trying to make in my times piece. It gives you a different filter to understand what's going on because it's a confusing moment because we have two things happening simultaneously at the small number of frontier labs that are heavily connected to this world. A, you're developing a complicated, powerful technology spending giant amount of capex on this, right?

So transformer-based LLMs are a legitimate breakthrough in AI. If you push them, put enough money into them, they're very jagged, but there's all sorts of interesting things that happen. It's a live wire technology. And so you have a real technology that's developing, but then it's being run by people.

who have these sort of futurist, almost apocalyptic sort of eschatological views, where basically I feel like that community split, where the hardcore original rationalists like Yukowsky or, you know, Nate Soares, et cetera, would argue like, you know, we need to bomb the data centers because we don't know how to control this yet. We're all going to die.

And then you have like the Altmans and the Amadeus who are like, I think what's more important is that we control the digital god. We get there first, right? So it's either going to redeem history or destroy us all. So you have people running these labs and they're hiring people who think this way. Andrew Ho online was talking about there's probably a lot of pressure within these places to think that way if you're hired there. They're thinking in these almost like apocalyptic terms, like we need to build the digital god before someone less trustworthy does, right?

Like the end of history is coming and we need to be the ones to get there. And you have this actual live technology that's powerful and a lot of money is going into. And so these two things are now mixed together. And I don't think our sort of public or media landscape was ready for that. Like this mix of weirdness with a real company and real technology.

And it's like these two things together is blowing everyone's mind. They're like, well, I don't understand. Like there's these people saying we're all going to die. Like, why would someone say that? Why are they so calm about it? And I think it's almost like the Rosetta Stone to understand what's really happening in this moment is that we have a mixture of a lot of money and a powerful technology with a weird group of people.

And that mixture is blowing up.

What does that manifest as? What is the specific thing that... people need to know.

My concern, all right, there's a lot of different things are going on here, but my concern is if you look at what specifically is causing trouble, what are we worried about coming out of the frontier labs? In particular, the sort of rogue AI narrative is what seems to have caught people's attention. The experiments that led to something like the Hugging face attack were extremely negligent, right? It boggles the mind. I've talked about it on your show before, Ed, and people point to that episode, so I won't go through the whole thing. But they're taking an agent as basically a prompt loop. It's a program that tells an LLM with a prompt. Hey, I'm an agent that has access to these tools. Here's my goal. What should I

do next? And then it gets a response from the LLM and it executes that updates at state and then says, okay, here's my new state. Here's my tools. What should I do next? So it just blindly executes what an LLM says, uh, which.

Could be okay.

Millions of programmers every day use agents. There's no problems with them, but they ran these, uh, over the summer open AI was running these negligent experiments where they gave them very powerful tools. that you could use for computer hacking. They hooked them up to an LLM that had no guardrails and what information they would give them. They made the programs persistent. So they said, we want you to just keep demanding an answer. Don't accept, you know, don't accept an answer of like, I don't know what to do next. And then they let them run for days without supervision. And they read thousands of them at the same time. And why did they run thousands of them at the same time is because what we now know

is what they wanted to do is see if any of these versions of happened to like stumble onto a particularly good hacking strategy, they would then use that trace to go back and reinforcement train the model to push towards that type of solution in the future. So it's like putting a thousand self-driving cars onto the interstate and saying, yeah, most are going to crash and maybe hurt people, but those that actually make it to the destination, we can use their data to help update our system. So it's this very negligent experiment they were running. And I think my claim is- I don't think they cared. And the reason why I don't think they cared is because they have this bigger eschatological belief of we got to get

to the digital god before anyone else.

And you think that that's what's driving open AI though?

Well, this is what I'm worried about is that when that's your mindset, instead of focusing on useful products, things that can give you return on investment, common sense safety standards, you're going to take big risks because when you're racing towards super intelligence, there's going to be some casualties along the way that's acceptable. But that's my concern. I don't know, Adam, if you agree with that, but that's my concern is because they are so convinced this is going to happen and it's so important and it trumps, this is what their writing says. It trumps any measly concerns we have about like the current moment that they will be reckless.

It's also why now Amadé and Altman are talking about, well, we need a global slowdown, whatever that means before we can stop doing dangerous things. Because if you believe that's the only way to get digital God, you're like, we can't slow down unless everyone else does. Because if someone else gets the digital God, It's going to enslave us.

Yeah, no, I agree basically with all of that. I mean, this is why they're doing what they're doing. And it's just awfully convenient, isn't it?

Right?

When you have this belief that this is inevitably what's coming, that this is what you are building and you know that you're right about it. Well, it makes it really easy to ignore any other concerns and especially any concerns about, you know, something concrete that's definitely happening as opposed to this.

I mean, the nicest way that I can describe their belief in this is that it's highly speculative. I think that that's too kind. and that it's a more accurate way to describe it is it's a religious delusion.

So, but could there not be? Because I don't know if I buy that this is everyone working at these companies, but I think what you're seeing right now with this slowdown talk might be a result of the schism between there are lots of rationalists and then there are other people who are working at computer company. I don't think you need that. That's why you've got weirdos like Jacob Coxon going out being like, I'm so scared of the computer without really saying what they're scared of. And also people being like, no, no. I love AI, of course, but this is cloud software. Because if it was just rationalists, I don't think they'd be so apologetic. But it kind of explains why Amadé is like, I think it's time

we finally tell the truth about this while also not telling the truth. or saying what they're telling the truth about.

I don't think you need that, Ed. I mean, I think if you are in this sort of rationalist philosophy, the slowdown is like, no, they're afraid of the digital god. But they're like, we can't stop if other people are going. Like, I mean, the more I understand about the belief system here, the more I discount, for example, the arguments about marketing or the arguments about this is a good way to make your technology seem powerful. I increasingly believe like, no, this is just what they believe. Like they started these companies for this purpose. Um, and, and I think it's the controlling atmosphere in those companies, not everyone at the, but so how do we know that?

Right.

I think it's a controlling atmosphere.

Yeah.

Not everyone else feels that way, but what do we notice with open AI in the last four or five months, they've had a steady stream of, uh, high profile hires from other segments of the tech industry that within like a week or a month leave.

Right.

So we've been seeing this revolving door. I have to imagine, you know, they're hiring you because they're at a size now where you have to hire executives that aren't from your own world. They're coming in and perhaps what's happening is they're hearing the rational rhetoric and are like, uh-oh, you know, danger Will Robinson. And they're out the door again.

Like, I don't want to be involved.

So you think that Jacob Coxon might be genuine?

Oh, I think Jacob Coxon is definitely genuine. Yeah. Look, he gave an interview to Wired a few days ago. Okay. And in that interview, this is what they used as the headline at Wired. They said, you know, that he was calling this crunch time for humanity. What they didn't say, what Coxon didn't say in the interview, or at least wasn't published in the interview is that is a direct quote from Eliezer Yudkowsky from like, you know, 15 years ago.

What was the exact quote?

That this is crunch time for humanity.

So the reason the only pushback I'll give you there is he uses the phrase crunch time and he says, And I'm going to pull the quote now. These are actually just literal quotes from my colleagues at Anthropic. The reason I push back is because all of this started with him doing an exclusive with the Wall Street Journal, which is not traditionally how that works. I used to run a PR firm. You don't get an exclusive with the Wall Street Journal because you left the company and you're scared without the assistance of a PR firm or at least an organization like EA. So I'm like, it feels like this is a co-opted thing, almost.

I mean, that might be true.

I don't get the sense. He can't even explain what he's worried about. Yeah. Well, no.

I mean, I think it's true that, right, you have this schism in the EA rationalist world. where you have the EA rationalists who are at the actual labs themselves, who are like, we have to be the ones to create the digital god. And then you have the giant infrastructure of EA and rationalist institutes that are incredibly well-funded that are saying, don't try to build the god at all because it's going to be really bad. So I think that's completely plausible that one of these, you know, obviously has connections to some of these EA institutes from before he took the job at Anthropic. So sure, that's, I think that just matches what we're saying. Right, so he is more from the camp of don't... Yeah, but

I think that there's.

A big difference between a young man taking an opportunity, which is genuinely what I think. I really think that that's what he's doing. We can disagree, that's fine. And the EA cause taking advantage of that, because like Daniel Kokolachev, whatever his name is, Kukuracha from AI 2027, quote posted it within like seven minutes. Yeah. It was very... clearly coordinated on some level.

But to what end? I mean, I've seen that, and it might be true. Maybe he's the Rosa Parks of the EAs or something.

No, I don't think he's... I think he knew that if he did this, it would get a bunch of attention. Because my big thing with this is, I think this is a burn-after-reading situation. I think that the vast majority of The vast majority of the problems caused by this situation are, there are people who genuinely believe this stuff.

There are people who cynically believe it, and then there are the rest of them who don't know who to trust, who are just going like, yeah, right? Like, sure, I agree. Like, this is really bad. Like, Sam Altman, I don't think he's a rationalist. I think he's a guy. I think it's actually quite funny watching Sam Altman adjust to this, because he's just like, yeah, um... Slow down safety council, I guess. What do you think? I don't really care.

Dario Amadei, now that you've said this, I will buy that he actually believes this stuff, which only makes the whole thing weirder. But I think that the mishmash of different cynical people, honest people, everyone's kind of weird, it's causing chaos in the AI industry right now. People are saying, oh, it's an op. Oh, this is all a plan. No, I don't think they have a plan at all.

Yeah. No, I mean, I think that's a fair way of putting it. um i mean like to me what was more important than jacob's tweet was the multiple responses from current anthropic employees are like yeah he's right yeah yeah so strange and it was so calm and i think that blew a lot of people's minds but i'll tell you who was not confused by that was probably adam right yeah.

No that's exactly what i expected yeah yeah Yeah, exactly. No, because, you know, there are a lot of people who believe in this stuff who work at these AI labs because they think that that's the best way to keep a handle on this is to be the ones who control the coming Godhead. And, you know, I understand, Ed, the impulse to think that, you know, that this is something that people are using cynically. And there probably are some people in the industry who are using it cynically. but there are an awful lot of true believers. There are so, so many of them. And these beliefs about what's coming have been so, so influential in the industry. It is the conventional wisdom about AI in the tech industry at

this point is that AGI is coming, super intelligence is coming, and it is going to radically and permanently transform the world. And it's going to come out of these AI companies working with these kinds of AI systems. There's just no evidence for any of that and a lot of evidence against it.

The thing, just to be clear about what I'm saying, actually, this is a point, I actually need to make this point very clearly. I think that there are a ton of EA rationalist types in this industry. I think Anthropic is full of them. I also think the overall industry led by Sam Altman played into their fear tactics and said, oh, we're scared of what this might do, insincerely. Some of them were sincere. Some of them were not. The outcome is they've created a very real panic about nothing. They won't talk about the real harms of LLMs. They'll talk about the big scary computer that might wake up one day.

Well, and I would push back on Sam Altman being insincere about it. That's just the way people talk in the world he came out of.

That's just how they talk.

That's how Elon talked. That's how they all... Elon was running all these safety conferences. That's how they all talk about this.

I'm serious.

Here's the thing, though. Using Elon Musk as an example of someone with a sincere belief, I've read those emails. These are just regular rich guys being rich. I will buy that Yudkowsky believes this or what have you. The rationalist types believe this, what have you. But it's like the idea that Sam Altman, it's just laughable. I just, I don't, that one, I don't really, I can't get my head around it. The same with Musk.

Well, in your defense, right? The defense of your claim there would be the fact that the EA rationalist faction of the OpenAA board tried to get rid of Altman, which was at the time widely reported as this must just be because he's kind of a slimy guy. But It's also true that specifically the people who voted to do this are strongly aligned with Yukowsky's world and thought that he wasn't serious enough about it. So that would actually probably back.

Up the idea. Yeah, and Helen Toner, who was a big EA person, she got kicked out by Altman.

Yeah, they cleaned it out. All the EA people on the board got kicked out, including– what's her name? God, I can't remember her name.

Joseph Gordon-Levitt's wife got kicked out.

Yeah, that's the person whose name I can't remember. Yeah.

Jesus.

But then– okay, so– Putting aside that sort of nuance, how else can we empirically test, you know, the effect of this philosophy on these particular companies? Well, I laid out a test in my piece. I said, here's two things we can do. Look at other companies in the space that aren't as strongly connected to rationalism, but are just as cutting edge and really understand technology and see how they talk about it. And I pointed to NVIDIA in particular because, you know, Jensen Wong is, he started that company while Yukowski was still a teenager. He doesn't have a lot of strong connections to that you know, Silicon Valley futurism. And I said, also you have to keep an eye on China. They don't, they have no

connection to that sort of Silicon Valley philosophy. And for the most part, they really don't talk about AI with that same sort of fear narratives that we do. Well, what happened this week? Jacob Croxon came out, they became a big discussion. What did Jensen Wong immediately said? He said, you know, that's dangerous and outlandish. So immediately he's like, that claim makes no sense, right? Today, China came out Like, what are you talking about a global slowdown? What is this like psyops, Cold War nonsense? Why would we start?

Why would we slow down our industry? Because they don't see it that way. They don't see their industry. They don't see AI as like a nuclear weapon or a godhead they're building. They see it as like an economic tool that they're trying to get trade advantage. So I feel like this week validated the claim that this particular philosophy is playing a big role. It's like that Sherlock Holmes story, the dog that didn't bark.

Right?

Like the thing that matters here is not the things being said by the two labs that we know are deeply connected to these philosophies. It's all the other people who are doing cutting edge research on this, who know it well, who are not coming out and saying, yeah, we also think we're about to kill all of humanity.

It's Nvidia, which is, you know, they supply all the chips to all these companies. They're intimately involved in these models and their evaluation. And I mean, they, they, They customize the API interfaces to their chips to specifically match with these. They know what's going on. And he's like, this is outlandish. What the hell are you talking about?

But what's funny is Jensen Huang is barely a week divorced from declaring AGI is here.

Yeah, but this was a bridge too far. This was a bridge too far this week, or last week, I guess.

So I'm kind of yes-ending, because what I'm saying is he has, as I think some have cynically, engaged in this Oh, AGI superintelligence wank. This conversation about it. And now suddenly it's like, oh, wait, maybe not that much. Maybe we don't superintelligence that hard. Actually, it's not superintelligence.

It's cloud software. Stop freaking out. And I think that this is the thing. It's regular greedy assholes locking horns with a cult, a religious cult of posters. Yeah.

I think that's definitely part of what's going on here.

What am I missing?

Well, no, I mean, I think, I mean, I just think that it's totally, totally possible that even some of the rich assholes genuinely believe this stuff. It's just that genuine belief for these billionaires, you know, manifests in a different way. because they believe that they have, you know, uh, like the power to warp reality with their money, which, you know, to a point is true. Like, uh, you know, uh, uh, Mark Andreessen is someone who I would never ascribe sincere beliefs to. Um, you know, like I don't, I don't, I'm sure he has some sincere beliefs and I just don't think that we have any evidence of what they are because I think he just says the stuff that he says because he thinks it'll make him money. But,

you know, he put out a manifesto a few years ago where he said that slowing down AI means that you are guilty of murder. Because... Cool. Yeah, cool.

Yeah.

Because, you know, that means you're slowing down the advent of a super intelligent AI that will save lives.

This is EA in reverse.

Exactly.

Yes. And... Jesus fucking Christ.

You know, that's a great excuse for... To say, well, but no, I have to make the investments that I make and you have to let me do it without any regulation or slowdown of any kind, because otherwise you're a murderer. I.

OK. As we wrap up, though, I was about to rant about these people. They sicken me. What should we do about this? As in, okay, maybe not the first thought in your head, but what is the way that people should look at this differently? How can they pierce the kind of rationalist veil and at least have a more sensible way of looking at how everything's happening?

I mean, I have a few suggestions right off the bat because these are the calls I've been taking all day, every day, it feels like for the last week. First of all is Clarity and terminology, right? Just as I wrote that op-ed, Michelle Goldberg has a similar op-ed that came out the next weekend. I think it's really important that people actually understand there is a particular philosophy that's heavily influencing these companies. At the very least, we should be factoring that in to try to understand what's going on.

There's a good quote in Adam's book from Timnit Gariboo, who basically said, and I'm paraphrasing a little bit, but it was referring to this issue. if it turned out that the major AI companies were run by Scientologists, we'd want to know what's the influence of Scientology on the decisions they're making or how they run their companies. That's what's happening. It's just rationalism EA instead of Scientology, but it's a very eccentric sort of almost cultish internally looking self-police type of belief system that's playing a big role.

We should at least know that's the case so we can factor that in. Two, we cannot allow the people in these circles to get away with co-opting the general term AI to refer to specific things they're doing that are causing problems. I see this terminology everywhere in the media this week, where it's all about AI is getting out of our control. We need to slow down AI.

AI is getting smarter and what's going to happen. We have to be very specific, right?

And I made this point in my newsletter as well, is it's the terminology of pacing the frontier is is in their terms. It's taking their philosophy about this. It's, we need to slow down. Everything we're doing is fine. We're just doing it too fast.

When in fact, LLM's not the right thing. This is a Max Tegmark mental model from his book, Intelligence 3.0. They have this idea of AI- Is.

That the future of life guy?

The future of life guy. Yeah, which is also deeply, deeply connected.

I've met him. He's nothing special.

But he had this idea. he had this idea of thinking about AI capability like a water level that rises, right? So that if anything is getting better, it leads you to think all of AI capabilities are getting better, which is partially why OpenAI is spending so much money on obscure math problems, right? There's not a financial, why aren't you building tools that like businesses will buy? It's because of the mindset of, well, if we can solve a math problem this hard, we can do everything comparably as hard.

So they want to connect it all together. But if we get more specific, No, the issue is a very specific thing. First of all, it's in the world of LLMs. Most AI is not LLMs. Within LLMs, it's specifically in the world of agents, but not all agents. We use, you know, millions of people use coding agents every day.

They never go rogue. There's no issue with them. Like we don't, we're not worried about that technology. So it's a particular type of agent, which I call over-equipped under constraint, where they give them very powerful tools. They run it with no constraints. It's a very specific type of system. that caused the hacking attacks and all of their concerns about recursive self-improvement is because they want to next take these over-equipped, under-constrained, LLM-powered agents and allow them to update their own code. So now they're completely unconstrained and unmonitorable.

It's that that we're worried about. So I think it's very important that we talk about specifically the thing we're worried about because it's very narrow. And then third, if I'm Congress, don't pass a bill about we need to, in some vague sense, slow down AI. Bring these people in to testify and say, tell me about these over-equipped, under-constrained, ill-empowered agents. Why are you running those experiments? Was there no way to run those safer? Is there other things you could, why are you focused on that? Is this what you mean when you're saying you're worried? Like, okay, what do you believe? Do you believe, and going through like Yakowski, like we really need transparency, I think, on this situation. That's what I would do, because I think

it would really change the dynamics If we could pick apart what's true about the technology, what is being hyped, and what is being driven by these beliefs and is causing more danger. Because you could shut down those over-equipped, under-constrained, ill-empowered agent experiments tomorrow. It would have almost no impact on your ability. All the products they're working on that are actually money-making, it should have no effect on their ARR, on their revenue, or anything. But they're not because they think that's the way to get to super intelligence. So that's what I would do. I would be super clear about what's really happening here. I'd be super clear about what rationalism is and philosophy is. And I would start

pouring transparency on specifically what they're doing and why, what they believe, what experiments are dangerous, why they're running them so dangerously. That's the hearings we need right now.

Adam, any additions?

Yeah, yeah. I think that those are all good ideas, but I think the thing that I would add is We also really need to try to shift the public conversation on this stuff, right? The response to, you know, Coxon, the whistleblower at, I don't even want to use the word whistleblower, the guy who left Anthropic and made a big splash recently. The response to that should have been, yeah, of course you believe that because, you know, you're part of this group that has these weird beliefs. Like it's a nothing burger. Like Cal said, I wasn't surprised by this.

You know, I wasn't surprised by what he said and what his colleagues and former colleagues said in response. I think that we need to get the message out there that this narrative about existential risk caused by super intelligent AI going out of control and that that is the most important and most likely thing that we need to worry about with AI. We have to get the word out that that is just a belief that comes out of a almost religious cult-like object, the effect of altruists and rationalists. This is not something that is based in reality.

And the real harms imposed by this technology are very concrete and very well known.

Awesome. That is great. I'm so glad you guys were here for this because I feel like at this particular time in history, as both of you have said, there is no clarity here. There are a lot of assumptions being made. And I'm really, thank you so much for joining me because I think this will really help people clear this up.

Well, thanks, Ed. I think it's the more we get this message out, the better. Oh, by the way, I would add one little extra caveat.

Don't look at X.

Yes.

It's completely caught up in this engagement farming off of this rationalist EA belief system. If you're a reporter out there who's listening to this, do not look at the X conversations. Call real sources. Talk to them in person because X is off the rails right now.

Indeed. All right, I'm going to end it there. This has been Better Offline. I'm Ed Zitron, joined by Adam Becker and, of course, Cal Newport. I'll be back with the monologue next, well, I think it's going to be Friday. Anyway, this Friday. Cheers, everyone.

Thank you for listening to Better Offline. The editor and composer of the Better Offline theme song is Matt Osowski. You can check out more of his music and audio projects at mattosowski.com.

M-A-T-T-O-S-O-W-S-K-I dot com.

You can email me at ez at betteroffline.com or visit betteroffline.com to find more podcast links and, of course, my newsletter. I also really recommend you go to chat.wheresyoured.at to visit the Discord and go to r slash betteroffline to check out our Reddit.

Thank you so much for listening.

Better Offline is a production of Cool Zone Media.

For more from Cool Zone Media, visit our website, coolzonemedia.com, or check us out on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

Stopping The AI Safety Cult ft. Adam Becker & Cal Newport

Kapitelklicken springt direkt hin

Shownotes

Transkriptvom Podcast bereitgestellt · Zeitmarken springen hin