Monologue: Jacob Coxon and the AI Safety Grift

Hello and welcome to this week's Better Offline Monologue. I'm your host, Ed Zitron. I've now had quite a lot of emails about the post by Evan Hubinger and Jacob Coxson of Anthropic saying that they, and I quote, earnestly believe AI could kill all humans, with a higher than 10% chance that it happens within the next decade. I want to lead by saying that these men, regardless of their intentions or cynicism, are beyond loathsome. They are disgusting to me.

I find them vile. And I find it equally vile how many people are falling for their garbage. If they truly believe what they're saying, the idea that they're continuing to work on a product that could destroy humanity makes them want to be war criminals. Coxon, who recently quit Anthropic, did so in an extremely public way that has been covered by much of the mainstream media. claims that he quit, to quote the Wall Street Journal, because he doesn't want to participate in an industry-wide rush to build AI systems that can improve themselves, which, as I will refer to later, is referring to recursive self-improvement, which nobody has proven actually is possible. In an interview with Wired, Coxon spoke at length,

without really explaining what it was he was scared about, outside of one moment where Max Zeff asked for specifics about his concerns, at which point Coxon incorrectly describes the hack as AI doing this all of its own volition, referring to the Hugging Face attack, only for him to respond when pressed about whether this is companies just moving recklessly fast that he didn't want to focus too much on the Hugging Face attack because there's plenty of evidence that the companies don't know how to align the models properly.

In other words, the moment that Coxon was asked to get specific about what the companies are doing wrong and where they're making mistakes, he punts to a non-answer. By the way, Coxon's solution to all of these problems is that the AI labs should agree to not rush towards recursive self-improvement. That's still a theoretical term for an AI that can train itself that everybody is talking about like it's real. In other words, Coxon is suggesting that Sam Altman and Dario Amadei put their clammy hands together and agreed to delay breeding the Grinch. We're not getting a Lorax, folks.

We're not making Shrek. We will not let the fairy godmother into your house, okay? Godzilla will not happen. Yet the most laughable part of the interview was, when asked about why he kept working at OpenAI and Anthropic for years, when Coxon said that it's also not hyperbole that we could cure cancer, because OpenAI, by stealing someone else's work, I'm not getting into it, it's all over the subreddit, solved the Navier-Stokes problem, adding that there really is no reason that we, referring to the AI industry, can't transfer that to scientific domains like biology. The kind of thing you say when you're just making shit up for attention and you're wrong. You're just fucking wrong. Those are two

very different things. A mathematical problem is not the same as biology. What are you talking about, Jacob? Why are the journalists just printing everything he says without pushing back? It drives me insane. Nevertheless... People have been saying, oh, oh, well, he's leaving because he's so concerned. He's got to tell everyone because he's so concerned. He's so worried. No, no.

The piece ends with Coxon saying that he'd like to do some sort of independent commentary on where things are going, which means a sub stack. And he'd like to do something like AI 2027 or move into some kind of accountability role. Yay. Yay. There's the grift. Wee. Congrats, everyone. Congrats on helping Jacob Coxon get a new job. Congrats. Congrats, everyone.

You did it. You helped elevate a real piece of shit. Okay. My dear friends in the media, my esteemed colleagues who allegedly have been doing this for years, Anderson Cooper, CNN, Wall Street Journal, all of you, everyone, how many fucking times are you going to fall for this? Why do you believe these people other than that some of them are saying vague and scary stuff and that they happen to be at the companies? Why do they never really say what it is they're scared of? And why, when they even try and get specific, not that they do very much, do they never blame the companies? I say it again. Nobody has actually achieved recursive self-improvement, nor have they shown any proof that it's even possible. But the

fact that Coxon is bringing it up in the run-up to Anthropix IPO, and as the rest of the AI hogs oink about it non-stop, makes it impossible for me to believe that Coxon has anything other than the most cynical intentions. People say, oh, he gave up his options in Anthropic, as if that matters in any way, shape, or form. He was barely there a few months. He probably didn't have that many and saw an opportunity to get a bunch of attention. He was at OpenAI for years and likely has a ton of options from there too.

Why in the world does this keep happening? This is how bubbles form. This is how grifters grift. It starts and it finishes with the media industry giving up on their jobs. It starts and finishes with the most important journalists in the world failing. And I think everyone failed here. And I am disgusted and outraged to see this happen again.

It's exactly what Matt Schumer did a few months ago with Something Big is Happening. It's the Citrini Mamma about the global intelligence crisis. It's the same shit. It's science fiction dressed up as fact. And the fact it works is only because, to quote Ed Elson, we have this cult-like worship of the wealthy where we believe that the people at the companies and the people will move around from these companies in a way that's only honest, in a way that's only true, when in fact these are some of the most cynical people in the world, which is pretty obvious in the fact that they go, oh, I'm afraid that this stuff is going to destroy the world, but they keep working on it. And I don't give a shit if he's not working at one of the companies.

If he goes into an alignment role or a non-profit for this shit, it's the same thing. It's a way to keep milking the cow. And Coxon's entire argument centers around this nebulous idea that things are moving really fast, but never really crystallizes on what it is that's moving fast. Nor does it ever hold the companies in question accountable.

He told Wired that it was important to ask executives on the record to give an actual probability for extinction in the next decade, and I want to be clear that this statement means absolutely nothing, and anyone taking the question or the answer seriously is a goddamn mark. What does the answer even mean? What's the difference between a 10% and a 30% chance? What does 10% even mean? What does any of this mean? Ah, who cares, put the AI hype in the bag. Who gives a shit, right?

Coxon claims that this is not a marketing stunt. By the way, the biggest sign that something is a marketing stunt is when someone says that. And that executives and senior researchers couch their phrasing in the press to sound sensible, which is hilarious because I've been hearing these kinds of threats for years. Dario Amadei, Wario himself, said to Axios last year, or maybe CNN, that there was a 50% chance of AI wiping out all of white-collar labor, or maybe it was a AI will definitely wipe out 50% of white-collar labor. The fact that those It's not really clear.

It's kind of the point I'm making. It's just saying stuff. In any case, Hubinger, who still works at Anthropic, followed up on his post, agreeing about the dire threats of AI, but adding that the risk from the present models is low. But don't worry, though. His worries were around superintelligence arising from recursive self-improvement, which is happening faster than we thought, referring to something that has not happened yet. To be clear, there is an actual threat from LLMs. Anthropic and OpenAI are using hundreds of billions of dollars worth of infrastructure provided by the largest companies in the world to brute force hack, using LLMs bouncing off of each other because they can't find any other uses at scale. These models

are doing exactly what they're trained to do, which would make everybody ask why the fuck they're being trained to do it. As I went into with Cal Newport a few weeks ago, this is not autonomous hacking by agents that escaped sandboxes. It's anthropic and open AI throwing unlimited compute to make LLMs that can, quote, do cybersecurity and having really terrible security practices and not being able to train them in a way that was consistent or safe.

And then they still use them. They still treat them as if, oh, they'll just work it out or just maybe they don't give a shit. Maybe they don't care. I think that's probably the most likely outcome here, by the way. To be clear, whatever anthropic and open AI models that were responsible for the hack are very, very dangerous, but not because of any sentience or magic or thought.

These are LLMs talking to LLMs to decide what they should do next, prompting each other again and again and again, and having unlimited resources to do so. If we weaponize billions of dollars to sink into the automated scripts that hackers have used in the past, we'd get probably much the same results and be sending people to jail. Somehow, despite years of these dire warnings, the warnings in question never include anything about what's happening today with really any specificity.

There are no scarograms about how anthropic and open AI have near-unlimited resources to commit to dangerous, reckless experiments that amount to felony hacking. There doesn't seem to be any interest in holding these people accountable. There doesn't actually seem to be any interest in stopping anything. It just seems that we all want to have a jerk-off theater around scary things that nobody actually wants to define. Even when Coxon discussed the hugging face attack with Wired, he framed everything in terms of helplessness, calling the LLMs an agent swarm and saying that one of the agents did this hack as part of a general strategy for understanding more about the grader, versus a piece of software with poor

security controls and unreliable automations pursuing a task in an unexpected way. Coxon's language intentionally minimizes any kind of responsibility or accountability on the part of the AI labs, choosing instead to put the blame on powerful AI that they can't understand. There really is no economic reason to do cybersecurity stuff with LLMs, outside of the fact that these companies are hitting diminishing returns in coding and that there are exhaustive troves of vulnerability data online for them to train their models with and do exactly what they've been doing.

Brute force hacking of GPUs has existed for over a decade, albeit with different techniques, and the innovation here is a direct result of the unbelievable resources handed to these two irresponsible, disgraceful companies. But China might do this is not an answer to why are American companies doing this, because this is clearly a situation where OpenAI and Anthropica built something they were aware would mindlessly bash its head and millions of dollars of compute against the problem until it broke through.

Make no mistake, Dangerous AI is already in the wrong hands. Those of anthropic and open AI. If a regular person did the hugging face attack, they'd be in jail, because this seems like, based on discussions with experts, a clear case of felony hacking. And if this wasn't the result of powerful AI, everyone involved would be in shackles. These companies act as if they're being forced to build these tools and run these experiments, when they're actually acting with complete autonomy, far more than they should ever have been allowed to have, with resources that I've repeatedly said and will say again are virtually unlimited. They act as if they have no responsibility or ability to stop their own experiments,

or monitor them, or really do anything but continue to feed them training data and watch what happens in awe, but also write long blogs about how they're scared, and also fucking hack people. Let me be very clear about this. Anthropic and OpenAI have repeatedly and flagrantly engaged in what appears to be illegal hacking of multiple different systems, and have done so using AI GPUs sold by NVIDIA and powered by infrastructure built for them by Amazon, Google, and Microsoft.

They are the ones making every one of these choices, and they will continue to do so every time we elevate the voice of a cynical grifter like Jacob Coxon. The people that work at these companies are willingly engaging in acts that, if not criminal, are morally and ethically bankrupt. And the people who keep giving these dire warnings about the dangers of AI never seem to give a shit about what's actually happening. Always keeping your eye on some non-specific harms in some non-specific future, all while saying that they alone are the ones who will protect you from whatever it is that might not happen. That they're blowing the whistle on companies that have paid them hundreds of thousands of dollars. And indeed, we don't know... And I

think there are actually very good questions to ask about what his compensation from Anthropic was and whether it's ongoing. I ask this because a lot of people from Anthropic are very supportive of a guy who just quit. A guy who just quit and is basically accusing the company of creating something dangerous that will destroy the world. Isn't that strange? To Coxon, Eubinger, and any other AI doomer, I have to ask, what is it I am meant to do with this information? What is it any of us are meant to do? Are we meant to be scared of you? Are we meant to sign up to Claude Max? Are we meant to invest in the anthropic IPO?

Are we meant to regulate this stuff? How? Why do you never say that? Why do you never say what it is we're meant to do to stop this? Why do none of you fucking people ever have any kind of suggestion or call to action or anything? other than some sort of self-serving or company-serving diatribe about how powerful and scary AI is. And if you hear any loathing in my voice, that's because I find you loathsome. I find what you're doing disgusting. I think you are cynical.

I think you're malcontents. And I think you're bad for society. And the fact you're being elevated is dangerous, but not for the reasons you're saying, but because it proves that our media ecosystem barely has object permanence. Congratulations on exploiting it. You're a fucking asshole. I think the answers to these questions are actually pretty simple. You don't actually give much of a shit, and you want some attention. If you actually feared this stuff and felt a moral obligation to tell people, you'd be specific about what it is we should be scared of and Indeed, tell us what we need to do next. Every time, it's the same sorry fucking story. Oh, we're building something so scary and powerful,

and we can't stop it. Nevertheless, we're going to keep building it for some reason we never discuss, but when it destroys humanity in some way we can't explain, you'll thank us, I guess. Great. Thank you, man. Thank you for taking up the airwaves. Thank you for distracting from the real problems. You guys are horrible. You guys are genuinely kind of evil. If you wanted to do something about this, you'd do something about this. If you cared, you'd care. You'd be saying, we need to regulate in this way. You wouldn't be doing this mealy-mouthed horse shit of, oh, maybe they'll stop doing recursive self-improvement, a thing that doesn't exist.

Weird that you don't talk about what's happening today. Weird that you don't talk about stopping the hugging face attack of the future. It's always the positive. It's always the positive in, even when being negative. And it's hard to see this as anything other than cynical marketing and attention-seeking from two guys who should spend more time working on building technology than posting on social media about how scary things are getting. I don't like them, I don't trust them, and the whole thing's a fucking grift. Next week, I'll be back with Cal Newport and Adam Becker to talk about the people that inspired this AI doomerism, the rationalists, and how to push back against their noxious bullshit. I'll

catch you then. I love you all.

Monologue: Jacob Coxon and the AI Safety Grift

Kapitelklicken springt direkt hin

Shownotes

Transkriptvom Podcast bereitgestellt · Zeitmarken springen hin