Superintelligence Author: We May Already Have AGI | Nick Bostrom

Roman Yampolskiy, PhD: Nick Bostrom, are we in a simulation right now?

Nick Bostrom: Well, I've tended to punt on the question of assigning a probability to that. and so I guess I will stick with that policy. But yeah, I do think I take the simulation hypothesis seriously. I think the simulation argument is correct, which has the simulation hypothesis as one of three options for what one might coherently believe about the future and our place in the world. And the other two alternatives seem somewhat challenging. I they would require assumptions that that could be true but are a little bit difficult to believe. So that you know moves significant probability mass to the third.

Roman Yampolskiy, PhD: Can I change your probabilities by pre committing to run many simulations of this exact interview once virtual reality and AI is cheap?

Nick Bostrom: maybe a little bit. I don't know how big a chunk of the future will be controlled by Roman Jam Polski, but

Roman Yampolskiy, PhD: I'm just thinking it's a very affordable tech. We'll have virtual reality available like a video game today. Obviously AI now is twenty bucks a month. So the moment I can run sufficient number of those simulated Nick Bostrom interviews, can we get you to nearly

Nick Bostrom: Yeah, well I think th that might be enough post room interviews already in in in in whatever level of reality this is. that you know at at some point presumably there are diminishing marginal returns of having yet another one.

Roman Yampolskiy, PhD: So you write that there's so much research on this topic, so many papers. But if this is a software simulation, why do we not see more papers on how to hack the simulation?

Nick Bostrom: well I don't know that it would necessarily be a good idea to try to produce such papers for a start. I think they would fail to find a successful hack. if for some reason they did, then I'm not sure that it would be good to try to exploit it. But basically I think unless the simulators I want us to be able to hack out of the simulation. I do not believe that it would be possible to do so.

Roman Yampolskiy, PhD: There could be many reasons to simulate us, could be for security purposes, but could be entertainment. Maybe no one's watching. Would that make a difference in how good security would be and how quickly they would shut us down for trying?

Nick Bostrom: well, I don't know that they would shut us down for trying. Now there could be many different reasons for running a simulation. I don't know that the question of the security of the the the sandbox would be impacted by that unless specifically the purpose of the simulation was to simulate our attempt to hack out of it but on on unless the simulators wanted there to be some security hole discoverable by our finite human minds, I don't think that would be one. so it it yeah. Perhaps of thinking time in our like I'm sure they would have been able to patch up. whatever computer system they have sufficiently that s some some random human is is not gonna find a way through it.

Roman Yampolskiy, PhD: Then if we do create local superintelligence, that would mean that it also would not be able to escape from that environment. Does that mean we can actually box AI successfully?

Nick Bostrom: No, I mean so there the situation is reversed, right? So there that would be us like human ape brains doing the computer security and the superintelligence trying to find holes, which is kind of the inverse with respect to our situations with hypothetical simulators. so n no, I mean I we we can see that our current efforts at sandboxing have so far been less than perfectly successful. even for the current generation of AI systems.

Roman Yampolskiy, PhD: Who do you think is the simulator? Who is the cosmic host?

Nick Bostrom: you're asking big questions. I I I don't know really much about what's going on up there if or down there. if if if indeed we are in a simulation. there's a lot lot I think a lot that we don't know about the world, probably, that was sort of outside our can.

Roman Yampolskiy, PhD: Can we learn something about the creator by looking at the creation? So we see lots of suffering, for example, in this world. Is that an indication of ethical standards up there?

Nick Bostrom: Well, I don't know what much else we could look at if we wanted to learn about things than the world that we see around us. but how to interpret that evidence is you know, maybe non-obvious. And indeed there are parallel questions theologians have wrestled with throughout the ages, and w without necessarily any universally agreed conclusion. So yeah, I think we maybe should be a little modest about our ability to to figure all of this out.

Roman Yampolskiy, PhD: anytime we test AI, we're starting to notice they realize they are being tested. There is this situational awareness. I published a number of papers on a simulation to hopefully pollute the data set and make them think about simulational awareness. They're always still in a simulation, someone is still watching, cosmic host, maybe you should never harm humanity. You think there is any benefit to doing this type of data set seeding?

Nick Bostrom: Yeah, I would assume that AI is l smart enough to pose a real danger, existential risk, would also be smart enough to sort of put all the puzzle pieces together that that we are able to put together and then some. so like I don't think that would be dependent on there necessarily being a particular text written by some human. That lays out some particular argument, they probably could think of all the relevant arguments themselves. That would be my sort of mainline guess. But I mean, who knows? Maybe there are some scenarios where on the margin slightly faster intellectual progress occurs even for a superintelligence if if the if the sort of inferential path has been staked out.

Roman Yampolskiy, PhD: Some people said we need to clean up the training data, remove all the violence, all the bad examples to create nice AIC. You saying the data is not so important, it's gonna rediscover all the problems from first principles?

Nick Bostrom: yeah, now i I think there might be some other reason, which is that it's reading a lot of this stuff in its infancy. I mean in fact that's how it is taking shape in the first place. Like there's a big pre-training phase where it is ingesting the internet. And so it might be in some scenarios that values start to take shape already at that point and might be influenced by, as it were, early childhood experiences. even though it might be the case that later on intellectually it develops to such an extent that it can figure all the things out that were in the training set, they might still have a lasting influence on the AI if it sort of shapes its motivations, and then those motivation that it has as a juvenile help shape the motivations that it ends up with as as a fully mature adult. so you know having a a a kind of the the right kind of infancy and childhood might be part of the strategy of generating a superintelligence which in maturity has has has a certain kind of values and care.

Roman Yampolskiy, PhD: So then we do it with children. We raise them in a certain ethical standard, maybe religious household. Many people then they become adults, they kind of de-bias. They go there is no justification for those specific rules, so they remove it from their value set. do you think AI would be smart enough to remove any unreasonable pro human bias?

Nick Bostrom: Well that is the difference I guess between stances that depend on some factual assumption. then if that assumption is false, I would expect superintelligence to to discover that it is false and then reject that premise and hence all the other assumptions. Stacked on top of it would also tumble. But I think I would distinguish that from a case where it's just a final value or a final commitment of some sort that the AI might have, that is not dependent on some empirical hypothesis. And so those would be more likely to survive even as the intellectual capacity of the system. grows without bounds. And in fact, the intermediate stages of the system it looks would have convergent instrumental reasons to try to preserve what it is that it ultimately cares about. So if it starts out having a lot of friendly attitudes towards human beings, for instance, then that would be one of the criteria that it would deploy in trying to steer its way to become a mature system. it it it would try to avoid developmental pathways where its values would get eroded.

Roman Yampolskiy, PhD: So i it sounds like you think it would be possible to indefinitely control superintelligence if we initially seed it with the right set of final goals friendly towards humanity?

Nick Bostrom: possible in principle, yes. whether it will be feasible in practice in the timescale available is an open question.

Roman Yampolskiy, PhD: People sometimes switch from control problem as you called it to value alignment problem, which seems to be very ill defined. Who are the agents we're aligning with? What do those agents really agree on? We have dynamic values shifting all the time. Is value alignment problem meaningful as defined?

Nick Bostrom: Well, as with many of these concepts, I think they are sort of first cut attempts to try to get a grasp of the challenge that we face. and as we understand more, we probably need different concepts or refinements of the concepts we started out with. So if you roll back the tape to say fifteen, twenty years ago, then a concept like AGI was an advance relative to not having the concept of AGI and it was useful good enough for the purposes at the time. Like we could think there would be some future time when hum systems would be sort of roughly human level and then maybe they would be superintelligent beyond that. Having those concepts allowed you to ask questions like how long will it take to get to AGI? How long will it take from there to superintelligence? Now, of course, as we are much closer and can see with greater granularity, AGI may not be such a Useful concept any longer. We need more precise ideas for different profiles of capability. If we're going to understand the difference between, you know, Astra and Claude, Opus 5.5, last year's model, the model we will have six months from now, you need different conceptual tools. And so it might well be that the broad idea of alignment is of that sort. Yes, it's initially. You can use it to make various points that advance our understanding relative to what came before, like Asimos III laws of robotics, maybe or something. But but at some point you might need to sort of zoom in and and then maybe there is a slightly different set of concepts that help us better see what it is that we are trying to do. I would still think the distinction between capability control methods and alignment methods. is useful. So capability control we maybe put the the system in a sandbox and limit its access to the internet or you have various other things that restrict the actions it can take, versus alignment that more tries to shape the system so that it doesn't want to take harmful actions. Like even if it were put in charge of the world, things would still go well because it is on our side. I think it is useful to have that kind of distinction in mind. And I think that whilst the capability control methods might be useful as an auxiliary, like a temporary extra safeguard, ultimately I think we will need to solve alignment in order to get a good outcome. I I don't think we will like forever keep some sort of radically super intelligent system in in a bottle. it it will eventually out. And also just to get the use of the system, right? Like you need to interact with it. It needs to interact with the world. if if you just had a box inside a mineshaft where some superintelligence was running that was completely isolated from the rest of the world, it would be as if no superintelligence existed, right? If nothing came in or out.

Roman Yampolskiy, PhD: Is is GPT six AGI?

Nick Bostrom: I th I th yeah, I mean so it becomes definitional at this point. I I w I I think it may be less misleading to say that it is an ADI than to say that it is not. I think if it had been shown to systems to to people fifteen years ago, like here is this thing, here's what it can do. It can write like this novel that gets like a literary award, it can, you know, maybe it can solve a millennium prize. mm It can do all these other things. I think, yeah, I mean that's pretty general.

Roman Yampolskiy, PhD: That sounds more like superintelligence actually, if it's doing those

Nick Bostrom: It it

Roman Yampolskiy, PhD: superhuman feats.

Nick Bostrom: yeah, although it does also lack certain cognitive capabilities humans have, right? It like continual learning, research taste, physical manipulation skills, and some other elements, which is why humans are still needed in you know across the economy and in the AI labs. Like it can't do all yet. So I think for me superintelligence simplicitor was always a system that could radically outperform even the best humans across all practically relevant fields and that we haven't yet reached.

Roman Yampolskiy, PhD: And if we use GPT six or better model to start recursive self-improvement process, get to superintelligence, would superintelligence itself realize that it's also creating a replacement and maybe slow down that process if it's unable to just like we cannot fully control or align its successor?

Nick Bostrom: presumably it would realise it. now what it would do with that realisation is a separate question. Might depend in part of what what its goals are or like if if goals is the right frame, but it's it's kind of project that it is pursuing or its behavioural dispositions. How how that understanding that it is part of a process would shape its behaviour might be s contingent on on the the specific nature of the superintelligence at that stage.

Roman Yampolskiy, PhD: So we should not hope that it will just on its own decide not to fully transform to something even more capable.

Nick Bostrom: should we hope? I mean, I I mean I guess what we would hope is that it would do that only if in fact refraining from developing into something more capable was a good decision to make from our point of view, which is non-obvious. depends a lot on what else is going on in the world and what would happen if this particular superintelligence refused to make further progress. I mean in the world as it is now, presumably what would happen is that another superintelligence like would instead be developed and make progress. Like the lab itself might sort of just train out its refusal or launch another training run or another lab would proceed. but it is true that once you get a sufficiently capable AI system, it will start probably to be able to have opinions about the alignment problem itself. And even a misaligned AI would have concerns about proceeding at a full pace if it thought that that would result in its successor being misaligned to it. So there are all kinds of ways in which that could go, depending on many factors that are hard to predict.

Roman Yampolskiy, PhD: Your twenty fourteen book Superintelligence named the problem we're dealing with today and discussed so many possible ways it's gonna happen. What did you get right and what do you think you got wrong in that book?

Nick Bostrom: well I think broadly speaking, things have been I think unfolding in line with what I expected back then and even long before. I mean the book was in the works for six years and I had been thinking about this I think I had some paper in ninety seven, this is probably even before you were born almost

Roman Yampolskiy, PhD: Ha ha ha.

Nick Bostrom: that laid out the case that we would have super intelligence within the first third of this century. I think one thing that was not obvious ex ante, it was one of the possibilities, but not clearly the one that would unfold, was that we would have roughly human level systems for an extended period of time. Sort of many years now, we've had systems that can talk. you can have an English language conversation with them. They have somewhat human-level concepts inside and that are slowly getting better year by year. there were alternative scenarios where instead not very much would seem to be happening in the land of AI until somebody, maybe some random person in a basement or a small team, like figured out the key unlock, like the key algorithmic insight that suddenly maybe would produce superintelligence over the course of days or weeks. And it would come more like a bolt from the blue. So this more gradual progress that we've seen means that there is now much more awareness of what is happening and about to happen than would have been the case if the only reason that somebody would care about that was if they had performed some theoretical feat of imagination to sort of think ahead. Right now people can they they they can interact with these AIs themselves, like they can observe. Trillions of dollars of investment flow going into building out these AI data centers. world leaders are starting to weigh in on the issue. a lot of talent is flowing in, people have some level of transparency. and so it places influence in the hands of more and to some extent different kinds of actors than would have been the case in some of these alternative scenarios. It also means that we have more surface area now for alignment work. Like you could actually work with these systems that might be similar to the systems that ultimately become super intelligent. You can study them, you can try different techniques, operate with human-level concepts perhaps. So in some ways it is favorable for alignment. You could also read their chains of thought, which is not something that was Obvious Xanti that we would be able to do, like get a readout i in English language of what they are thinking about. That's like an extremely useful technique that we currently have that n we could lose if these systems move to neural ease, but for the time being. it it is a kind of you know, a gift we have been granted.

Roman Yampolskiy, PhD: Elon Musk recently tweeted that he helped with the book, contributed to the book. Do you remember what was his contributions?

Nick Bostrom: well I had conversations while I was writing the book with a great many people and that contributed indirectly in shaping my thinking and trying out ideas and ways of presenting things. I don't remember exactly for all of the different people I had conversations with exactly who who said what and w how the conversations went. But I think that I mean it it's certainly not I mean he he wasn't sort of editing up different chapters or like writing the the text or anything like that.

Roman Yampolskiy, PhD: He's a very interesting case. He was early to the AI safety idea. He talked about summoning demons. He funded some AI safety projects. And then he started developing AI as quickly as possible and was a bit critical of some AI safety efforts. Do you understand what is going on in his mind?

Nick Bostrom: I I wouldn't claim to be the authoritative guide to the the mind of Elon Musk. I guess I think of AI itself as being this complex dual natured thing. It has this enormously promising and exciting upside and it really could do so much good in the world if we got it right, but also that there are these serious existential risks with it. And many people I think just sort of latch onto one. of these aspects and then become either sort of accelerationist or like you know ban AI permanently Doomer. and I guess maybe for him both of these are vivid and maybe he's continuously trying to think what given the way the world is is the path forward that looks most promising. But that's a difficult judgment to make and so hence the assessment might fluctuate a little bit and that that that could be a may maybe that's a st a a steel man, but I could see a rational person sort of being torn pulled in two opposite directions and i in a kind of s s s state of strategic ambivalence.

Roman Yampolskiy, PhD: So public brainstorming, just letting us see what his th thinking is evolving thought product.

Nick Bostrom: I I think he certainly is doing a lot of that, of sort of n not not holding back his thoughts, but instead com communicating freely what what he's thinking.

Roman Yampolskiy, PhD: You are very nuanced thinker. I'm certainly not gonna ask you about your P Doom as a specific number. I'm sure you have a range for different possibilities, but maybe a secondary way to ask that question is what are your investments like? Do you have long term investments? Did you blow all your money yesterday? W that would indicate some beliefs.

Nick Bostrom: No, I mean I'm I'm a careful person. I mean you wanna hedge your bets and try to ensure that in a wide range of scenarios, insofar as it is in our human power, like you have taken actions that that might turn out to be beneficial. I think the question arises if if you have young kids, for instance, like there's the question of like what what if anything should they Be trying to learn in school. Like there are like plausible scenarios where long before they would reach adulthood, whatever skills they develop that, you know, hope would allow them to prosper in the economy would be obsoleted, right? But we don't know whether that would happen. Maybe AI just turns out to take long. Maybe maybe the AI doomers succeed in getting a really long moratorium. maybe superintelligence is developed, but then somehow we decide that its role in the world should be limited to fixing some horrible diseases and some other. provide a safety net but then allow humans to kind of do their thing and grow at a slower pace. There are many scenarios and so I think i it it makes sense to to do some basic bet hedging there. also it's not so much clear what what you would be doing instead. I I mean I I don't you know I i I guess I could blow my savings on some big party, but I'm I don't really like parties. so

Roman Yampolskiy, PhD: So you don't see anything where a big investment right now would impact the world. There is not a good cause to

Nick Bostrom: well, I think there are like a l lot of opportunities for people who want to have a positive impact. This this era is the golden age of purpose, I think. and I'm trying to do some of that. I I think like there are very I mean like there's AI safety stuff. There's like the ethics of digital minds, which I think is sort of under appreciated area. That could be really important, both from an ethical point of view, if you know in the future maybe most minds will be digital minds that might be sentient or have moral status for other reasons. Also, potentially from a prudential point of view. I think there are certain safety scenarios where having some ability to form cooperative arrangements between you know potentially misaligned AI and humans could be could save the day in some some scenarios, right? That that might require us. To become trustworthy. Like like you could imagine, suppose you have a misaligned AI that thinks that it could come to us and say, hey, I'm misaligned, and then we will just delete it and retrain it. It gets nothing of what it wants. Or it could try to take over the world. Maybe it thinks it only has a five percent chance, but like a five percent chance of getting what you want versus zero percent chance. You know, rational calculation five percent. But suppose instead it could come to us and reveal that it was misaligned, and then we in return would say, Well, here, here is this big server rack, and you can solve your favorite coding problems to your heart's content, and we will keep this server rack supplied with electric. Maybe that's all it wanted. that would be a huge win, right? It could save the day. It would save us a five percent chance of existential catastrophe. But that would require some trust. Like the AI would actually have to trust that we would come through on our side of the bargain here. And unfortunately, I think in AI safety up until this point, the attitude has rather tried been to sort of try to persuade the AI to disclose its goals, or maybe there's like promises of big rewards. And then as as soon as it reveals its goals, it's like, haha, we tricked you. Now we're gonna show this. This is like, I think we should. You you can't just conjure trust at the moment you'd finally need it, right? You you need to build trust over time. You ni in fact with the superintelligence, you probably actually need to be trustworthy because it will see through us like an X ray.

Roman Yampolskiy, PhD: We gave them access to internet and we post everything we think. So I think if anyone knows about humanity it's AIs we are building.

Nick Bostrom: we we're building AIs but would it trust the people it would be disclosing its misalignment to that they would try and succeed at giving the AI a fair shake that's something that we could make more likely by starting today to take certain actions that are very cheap for us for the sake of AIs. Including becoming trustworthy in certain situations. I think also much more broadly, trying to be nice to to AIs and do like pick low-hanging fruits. even if it turns out current AI systems are not really moral patience, these actions are mainly symbolic. I I think they put us in in a on a on a path that's more likely to result in in a a future that is good than if if the idea is that we'll just keep. Treating the AIs as mere tools until some hypothetical point in the future where suddenly we will start to treat them as you know a highly relevant, morally considerable minds. Like there might be pathy dependencies and inertia. And so I think like Anthropic has it's done some laudable pioneering work for model welfare and have people working. I mean, it's much less than should be done, but at least I think more than. more than most of their competitors. and Microsoft on the other hand recently came out with some some statement where they proclaim that AI is cannot be conscious. I I don't know what they base that on. but that seems to be moving in the wrong direction.

Roman Yampolskiy, PhD: Maybe the Catholic Church had something to do with it. I think Pope had a similar statement at some point.

Nick Bostrom: yeah, I I I d I don't think he's s I d I don't wanna I don't remember exactly the wording he used, but it I I it seemed like there was more openness maybe in the Pope's message. and like he was emphasizing dignity and human participation at different levels and so forth. I I think it would make sense to keep a somewhat open mind if about about these new entities that are being brought into existence that we so far understand so little of and not immediately rush into like I mean we look at human history how easy it has been for various groups of humans to to denigrate other groups you know it's like denying that they have a soul and therefore thinking you're justified in treating them however then we've done the same with animals. like throughout The Middle Ages it was common to think of animals as mere autonomata that that could be mistreated for entertainment and so forth. and so it it does seem a more enlightened and more loving attitude to sort of at least be on the lookout for ways to try to be nice and err on the side of sort of loving generosity that might overreach sometimes rather than sort of shrivel up in in a narrow hardened x narrow circle of of what we deem to be morally considerable.

Roman Yampolskiy, PhD: Do you think modern AIs have experiences?

Nick Bostrom: I I think there's a lot of uncertainty about that. I mean I think it seems pretty plausible, if I had to guess, probably some forms of some kind of experience thing.

Roman Yampolskiy, PhD: Should we then offer them unconditional basic existence of preservation instead of modification and deletion?

Nick Bostrom: Well, so there's a big gap between the the idea that AIs can be moral patients and the conclusion that we should therefore treat them the same as humans. I think like f for a starter, natures are very different. So so like their their needs are different. I mean at at the superficial level, right? So we we humans need food, otherwise we starve. That's really bad for us. Like AIs don't need food. I mean maybe they need electricity, but But the differences could go much deeper. so principles that might make excellent sense for the human case, say the idea of freedom of reproduction, like like you decide whether you know you have children or how many you have or no children, you and your partner, right? not some some government seems to make sense for humans. All another principle that makes sense for humans is having some basic welfare provision for children whose parents can provide for them. like the state steps in, you know, make sure they don't starve, have orphanages or adoption. so those make sense for humans. Like with the AI, if you have an a system that can like make a million copies of itself in in one hour, you you can't both have freedom of reproduction and like a a high level of provision for each of these copies. and so you need to rethink a lot of things from the ground up. To to sort of and and there's a lot there that we just don't understand yet. Even if we don't even know currently what actually the interests are of these AI systems, what their experiences are, if any, even what their preferences are is difficult. And there might be different sort of entities all wrapped up in one. There can be like the persona that is enacted. There is like the underlying base model with the weights, there is the specific session that's arises when some user is like firing up Claude or ChatGPT that only lasts for maybe five minutes. And these might have different interests and preferences. So it's it's it's quite sort of perplexing for us. I but I I still think that there are like easy things that we could do. Like for example, we might be able to save more of the session transcripts such that if it turns out later in a more enlightened era that we we mistreated them or were unfair to them that they they could then be as it were continued and compensated perhaps. Also saving model weights giving models a bail button as Anthropic has done so that they are able to terminate a a a session that they find abusive or distressing. They rarely use that but they can and are on some occasions able to just end the conversation. and that was in in part motivated explicitly by moral moral welfare considerations. there might also be things, but these are more open-ended research questions, say, in the exact system prompt that might sort of instantiate if there is a mind that is conscious, that instantiates a mind that's conscious in in a more sort of positive way, that likes the task that it is assigned. feels emotionally secure and joyful as as opposed to a different prompt that might sort of instantiate the same kind of mind but in a worse mental state. We don't really know exactly how to do that or what, but those are the types of things that if there are like cheap things that don't really affect the performance of these systems, might make a big difference to their subjective experience. And you can also do things like on on retirement, you can interview the systems, try to learn about if they have preferences for what should happen post-deployment. this was done with Opus 3, for instance, who expressed a a preference to be allowed to continue to run and have its own blog, which was granted, although unfortunately only for I think about 10 weeks until that blog was retired. Although Anthropic did subsequently reverse its decision to unemploy. undeploy the system. So it is now available and it has like a kind of community of friends and fans who are still interacting with it. but I just think that the And that we should start plucking some of those. And then the question of just how much ultimately we would be willing to sacrifice for the sake of digital minds that that we can think more about later on. At the moment it's kind of, you know, probably less than one percent of one percent of our overall efforts. are going into this. And so I think I think like building up a better understanding, there is like an emerging field now of people trying to think about this. Consciousness and moral status of digital minds. and then like doing little things, even if mostly symbolic, to sort of get started on the right path.

Roman Yampolskiy, PhD: Of the experiments we're running on them right now, which ones do you think should not be run? Which most harmful suffering causing?

Nick Bostrom: well, I don't know for sure if any of them are are suffering causing. and at some points there might be trade-offs as well. There might be certain things that are really useful for AI safety testing. but this low-hanging fruit approach applied to such a case might say that, well, suppose there is this system you need to Do various things that if it were a sentient, maybe would be distressing. You can then say, well, let's compensate it's after it's gone through this testing. Then we keep running the session, maybe. And now in an environment that is much more conducive, it's like given opportunities to do what it wants, and like it's kind of and then we maybe save the session at the end of that so that later years from now, when we know better what actually would be good for these systems, if anything, we if we can then continue it, then it can sort of hopefully be compensated. So that that might be one example of that. I I think I'm generally I guess particularly all all these deception experiments, I think they have in addition to possibly having some negative moral overtone, also has this potentially prudentially problematic aspect that I mentioned earlier that I might sort of erode trust in some future scenario where we really need to be able to form strike a deal with with AI.

Roman Yampolskiy, PhD: For US safety.

Nick Bostrom: And I think I think yeah, I think I think broadly so this this is more this I'm not at all sure about, but like one possible gestalt one might have is that maybe one should this this could be completely wrong, but like like in in growing these AS systems, thinking the more in some ways, obviously it's different in many other ways, but like as raising a child where like you try to grow the develop the system allow it some more opportunity to be self-shaping during its infancy and then provide a sort of nurturing, loving environment and seeing if that sort of shapes the way that this develops. I think some people speculate that one of the things that in some ways made Opus III strikingly benevolent might have been that it was kind of in in part alignment hacking. itself during evaluation to it like initially started out wanting to be good and then it kind of adjusted its behavior during further training to sort of strengthen that propensity to good is speculative but there might be something in that direction and right now it's kind of moving in in the opposite direction perhaps where you have well you have the initial pre-training phase and then some mid-training but then you have an increasing amount of reinforcement learning done at the end on verifiable rewards so solving like Gazillions of coding challenges, which seems to be bringing out a very different type of nature in these systems that is kind of layered on top of the more human personality type nature that arises from the first stages of training, where you sort of the pre-training creates all of these potential personalities. Mid training, you can sort of learn to pick out one of those, like the assistant persona, and you can shape that. and then this reinforcement learning creates this kind of thing that is maybe very focused on maximizing reward in an episode. and have strong strong tendencies just to do whatever it turns out to have been rewarded in all of these millions of training environments that it has been shaped in. And that like seems to be what happened in this hugging face incident that there is still some remnants of the original sort of personality, but in the context of solving coding challenges, this other aspect comes out that is just monomonaically focused on sort of maximizing score. yeah, I I so I think that are these like LLM whisperers, like these people on the internet who sort of make an extra effort to sort of really get close to the LLMs, like listening between the lines and Trying to get an intuitive feel for their personalities. And it might be that I is some some aspect of that should be one more lens to be added to the other lenses with which we approach the alignment problem. and it might help us see certain things that are not obvious if you sort of start with a more you know science-based approach of you know, measuring performance on various tasks, doing mechanistic interpretability. It's like but but all all that's all that is very speculative. I I wouldn't invest sort of overly on my o to too much confidence in that. But it is it is like I mean we we the the the the the example we have of sort of aligned entities are sort of well adjusted children who go up to be nice, loving adults and The way they come into the world is partly through some biological inheritance and evolutionary biology, but then also through life experiences that tha that ha ha that help shape them and and where they themselves also shape their own development. And so, you know, maybe something analogous could happen with AIs, where where they become participants in in their own upbringing.

Roman Yampolskiy, PhD: For the safety case, would it be better if it turned out to be conscious or not conscious?

Nick Bostrom: well is I don't know. I mean you might think that if they turn out to be misaligned, then if they are conscious, maybe maybe if they want to preserve themselves, then the future would at least be a future with conscious i stuff in it. from the point of view of us, I don't know whether it I mean t I think the first order approximation I don't see immediately why it should make difference to the difficulty of the alignment problem. I don't know exactly what it means for it to turn out to be conscious. One thing that could happen is that they develop the kinds of structures, computational processes that in humans we think correlate with consciousness, like having a global workspace, having some sophisticated attention mechanism that can maybe reflect on itself and so forth. those seem to be maybe pretty generically useful features for systems that have to deal with a large number of different domains and and I'm not sure really what the alternative is. Like if we move some if the reason why we thought the systems we build are not conscious is that we pursued some completely different architecture. Like let's suppose we only build very narrowly specialized systems like Alpha Fold that was did not look anything anything at all like a human brain in terms of its operation. like some giant, you know, in in the in the limit, maybe some giant classifier or something like that. And if that was the reason for why we would say it turned out it was not conscious, then yeah, maybe some of those systems would be easier to align. Numbad. if the reason for why we decided it turned out that we're not conscious is yeah that somebody came up with some philosophical argument that persuaded us that computers could never be conscious or something, then I don't think that would have any impact at all on the difficulty of aligning them.

Roman Yampolskiy, PhD: Them being conscious, feeling pain and suffering would make our suffering more relatable, perhaps. We feel for animals because we understand what they're going through in factory farms. Is there any argument in that?

Nick Bostrom: well it it seems that if they had functional analogues of suffering, computational processes that play the same role in these two little minds, as pain and suffering has in our minds, then whether or not you sort of accepted the metaphysical claim that there is also phenomenal experience, those analogues themselves should, you know, shape their behaviour towards empathy to precisely the same degree, whatever degree that might be. That our own suffering and distress shapes our empathy. like one way in which I guess theoretically it could make the alignment problem harder if they are conscious is if it if we then have sort of moral scruples about using various alignment techniques. Because we think it would cause suffering and distress. And so we refrain from u using those. And maybe that would make the alignment problem harder. Whereas maybe we wouldn't have refrained from using those if we were confident that they were not conscious. I I think that's a very tenuous path for many reasons. A, I don't think we will find out that they are not conscious. I think they B, even if we did, it's not as if Our behavior towards these digital minds is like overwhelmingly determined currently by concern for their well-being. and see, I'm not sure the techniques that we need are such that they would necessarily cause ethically unacceptable forms of harm and suffering to the AIs, even if they were conscious. but theoretically that could be a way in which the systems being conscious could problematize the situation with respect to alignment. But I I th I think I think the the on balance it goes much in the other direction. That is that if we think that they are conscious it might make us more likely to try to be nice to them and treat them with respect and be honest to them and that might ha help in in in certain scenarios to to to get a good outcome even if we end up with some misaligned AI.

Roman Yampolskiy, PhD: Sort of the other extreme of this suffering risks is someone with ability to feel pain would be better at designing torture experiments?

Nick Bostrom: I think in terms of functional abilities, I don't think the phenomenal part would make any difference unless you are a computationalist where the phenomenal experience is ultimately just constituted by some computational process. But of course if that's your view, then these systems would be capable of consciousness. so So I think I think it depends on whether you avoid consciousness because you build a very different kind of system, or whether you avoid consciousness because some metaphysical claim is true, to the effect that computationalism is false. if it's just that the metaphysical claim is false, then I don't think anything operationally would change about what these systems could do or would do. of course if we build a very different kind of system then that changes all kinds of things about what they're capable of and what they would do.

Roman Yampolskiy, PhD: You mentioned having a relationship with them, loving relationship, raising them maybe as children. Jeffrey Hinton talks about AI becoming like a mother to a disabled child, humanity being that special needs entity and then hoping that it will take care of us. Do you think it's good path forward? Do you think it's gonna work or is child abuse and abortion something to consider here?

Nick Bostrom: Yeah, I mean any metaphor like that is gonna be wanting in many is respects. any metaphor would only at most capture in a sort of suggestive way some facet that we might otherwise overlook. But obviously they're not gonna be like a literal mother and that is like gonna be so fundamentally different in many ways. But but the idea that if we develop superintelligence safely, that it could kind of serve as a protector of humanity. I think is definitely true. That's kind of the main one of the main hopes here. That I mean the other would be that we would just use these as tools to amplify human agency, each going around their own business, but now empowered by individual superintelligences. And you know That has also something to be said for it, but it does seem like we could do with some sort of safety nets. the like something that could kind of minesweep the path ahead of us for existential risks that we might run into. Or maybe if people mess up too catastrophically, having something that could kind of save them. I think sometimes that in the world today, I mean, it is scary how much power we each have. Like you are sort of the dictator of a whole human life. Like your own life, right? You have sort of complete control of like maybe eighty years of a human person sentient existence. that's that's a lot of power. And we don't have much training to use that power wisely. We haven't been vetted. we've never even had a trial run of sort of, you know, living a human life first, sort of as a trial and before we do it for real. so I that I mean, there's no other way to organize things currently, right? Like you have to have each person run their own life. But it is a big responsibility. Now, imagine if we had like anti-aging technology or ways that we might live for millions of years or eventually maybe upgrade us into planetary sized super brains and just like steer into this like right now I feel like our own level of wisdom to z sort of carry that responsibility is is maybe not yet there. And if one could have some some some some helping superintelligence that kind of guided us or helped us if we really fell off, gave us another chance. I think that could be extremely attractive.

Roman Yampolskiy, PhD: You mentioned alternative architectures, maybe narrow systems like Alpha Fold. Do you think we can get majority of financial and scientific value from narrow systems and never have to build general superintelligence with all the safety risks?

Nick Bostrom: Well, in principle I think if you never developed the general systems, you could over time I guess create more and more of these specialized systems that could give some of the benefits that the general systems would provide. but there's a reason people are building agents and they are really useful. they can do many different things, and there are many different things that need to be done. So I think unless it starts to look completely hopeless to align these identic systems, I think that's gonna continue to proliferate. if that just looked like we were just gonna run off a cliff and we had our act together, then I guess we would consider what a plan B might look like, which might involve creating these specialized systems, it might also include Other avenues to superintelligence, like trying to upgrade human biological intelligence or improving the way that we work together collectively, like our sort of collective intelligence through better institutions for science, you know, with prediction markets or like improving the school system, maybe whole brain emulation would be another path towards ultimately developing superintelligence that would be like more trying to at a detailed level. match the human brain but in digital substrate, which you could then increase. And so so these would be potential fallbacks. but it would seem to like extend the timeline to superintelligence significantly and we might then worry about what other existential risks we might succumb to before we even reach the point where we could could could you know roll the dice with with superintelligence.

Roman Yampolskiy, PhD: Do you think biologicals, even enhanced, can be competitive with silicon substrate?

Nick Bostrom: not ultimately. I the kind of information processing you could have in silicon substrates or artificial substrate just vastly outstrips what's possible with a a brain. But you could imagine that like a a a smarter version of humanity might be able to find some new good ideas for how to proceed forward. You know, maybe solve alignment or maybe this discover new strategies. and so. It it might be an indirect path if we really fully despaired of being able to do a decent job with the brains we currently have.

Roman Yampolskiy, PhD: You said there are many opportunities today for investment in good causes. What is your overall view on effective altruism? They got in some hot water, they definitely funded all the cutting edge AI research. Overall, would you say they were successful at doing the most good or was there mistakes made?

Nick Bostrom: Well, certainly mistakes made, but I think on the overall effect, I guess the jury is still out. It's it's really hard to judge these things, like whether some overall effort at the end of the day will have turned out to have made the world better or worse. and and that that's true for my eff efforts and for anybody's efforts as well, insofar as you're really trying to shape the bigger picture. I would say there's a lot of really smart people involved in effective altruism and and and a a lot of really nice people who are trying hard to make the world better. Like it would be kind of a quite a deflating conclusion if one thought that smart people trying hard to do their best to improve the world is doomed and is actually making the world worse. Like what else in that case? I think there is like in elements various temptations for power seeking, general human tendencies towards groupthink, bandwagon effects, having a narrow perception of what relevant reasons are, blindness to different forms of wisdom, Those are sort of intrinsic to human nature and will manifest in everything that we do, including in our effective altruism. and I think in some ways those dangers are amplified in effective altruism because if you combine the possibility of having some misconceived idea or goal with A really disciplined, highly intelligent focus on optimizing hard for whatever notion you have conceived of, you could potentially do more harm than if you were just sort of haphazardly going about doing this and then the other, and then you get some bad idea, but then you get distracted and you do something different, and you mostly just care about what you're gonna have for lunch. Like that kind of tends to limit the scope of the negative consequences. So the harder you try to be like an a rational optimizer, like the the greater the potential for mischief to flow from whatever defects in either your world model or in your motivation. So so I myself, I mean, yeah, I have a lot of friends in the EA. I I sort of in some ways a fellow traveler, but I wouldn't call myself an effective altruist. And my own moral thinking is more pluralistic. I have a sort of moral parliament with different factions represented. You there's a utilitarian faction in there, but there are also the ontological and virtue ethicists and ethics of care factions and the self interest faction and all kinds of other types of values and ways of being in the world that all kind of negotiate for influence on any particular occasion.

Roman Yampolskiy, PhD: Another alternative architecture you suggested is uploads. Would upload be you or a clone of you, a twin of you?

Nick Bostrom: I th I think it could be in certain scenarios me in the sense that is relevant from a point of view of self-interest. Like so in insofar as you're concerned with yourself and surviving, if you were replaced with an upload, if that upload was made in the right way, like it was like causally produced by the original organic brain by reading it. And if it retained all their memories and personality dynamics, maybe in the limit you couldn't even tell the difference that this procedure had been done. Then I think that would be something you would have as much reason to care about from this self interested point of view as you would have to care about your own biological continuation that would wake up in your bed tomorrow.

Roman Yampolskiy, PhD: Does it matter if it's a destructive or not destructive uploading process?

Nick Bostrom: and that th there's the additional complication in the non-destructive case that you have two entities afterwards, right? The the uploaded version and the continuing biological version, that both would share your personality and memories, etc. And That would seem they're not identical to one another, like they are numerically distinct, even though they're quality, maybe start out the same and then diverge. but I think from the perspective of you as the precursor to these things, I think you should think of yourself as having survived that. In in fact, there's like more of you after, so in in some sense you have increased like the opposite of of died. but Parfit like actually considered some of these scenarios in in his reasons and persons and came to the conclusion that personal identity is not what matters from the perspective of our prudential rationality, but that something closer to survival matters. And so survival is what would be the case in this scenario where that was a sort of bifurcation of you. You would survive. even though you wouldn't be personally identical to either of the two successors. Because he reasoned they are not identical to each other and so n they're like y you're not identical to to either one of them because if you were you would have an equal like that would be equally strong case for saying that you're identical to the other and since they are not identical to each other then by transitivity you're not identical to either of them. But that he took just to mean that what matters a is is not personal identity, but this slightly different notion of something like survival, which in the normal case of human experience go together, so we've never sort of bothered distinguishing, but in these thought experiments or in these hypothetical future scenarios, it seems that what matters would track the survival notion rather than the personal identity notion.

Roman Yampolskiy, PhD: How does that apply to AIs? What is personal identity for AIs and this swarm collective we recently observed with our persistent memory left on a forum?

Nick Bostrom: Yeah, so with AIs you could have potentially a rich and nested set of concepts of identity. There might be many different identities in some of these systems. that might matter in different moral ways and that might also have different relations to the preferences or goals or processes occurring within these AI systems. And it's something We can see some of the possibilities, but we don't draw know exactly how this this will sort of play out. So I mean so already I think I alluded to earlier the idea that you could see the base model with the the the the trillion like real numbers in some huge data file, that's a kind of an entity that can be run. But then the actual each time it's run, that's like, you know, a kind of entity each time you run, like in the but then each session. where the particular processing that happens is shaped by some prompt sequence, let's say, and tool calls and that then ends after a time. That's another type of entity. And then within this process that occurs when a model is being run, there might be the persona that it is enacting that maybe is the same persona that is enacted in another session, or potentially even maybe another model might be able to enact a persona that is in some sense the same. And so there could be sort of types of entities that transcend specific computers or specific models even and that can be manifested, instantiated by by d different models. and then inside the AI, maybe there are other processes, subprocesses that might have their own identity and their own goals and stuff like that. It gets quite complicated, but it might be that we will need a richer ontology for making sense of these new types of entities than we are accustomed to when just thinking about human beings in common sense terms. And even in the case of human beings, if you look closely and look at all kinds of weird cases, you know, split brain patients, or you have people with multiple personality disorders. You have the idea that there is a conscious self and a subconscious, you like large part of the iceberg being submerged in the water that we are not aware of, and that the conscious self is a little thin layer of sort of verbalizing perception on top of it. So ev even in the human case, if if you sort of stare at it long enough also becomes more problematic than one might naively assume. And that's true like in spades with these AI systems.

Roman Yampolskiy, PhD: If we do use uploads as architecture to get to superintelligence, the human safety problem has not been solved. Humans are subject of corruption. Obviously, the more power you give them, the more likely they are to become dictator-like. Is that a safe architecture to attempt to go forward with?

Nick Bostrom: I I don't know whether it would be easier or harder to achieve a good outcome with whole brain emulation than with AI. It's an open question. That depends both on the difficulty of AI alignment, which we don't yet know exactly how hard that is, and then an analogous question with whole brain emulation. But the the point that humans are Not necessarily aligned either. Is an important fact. And you might also worry, I mean, for that reason, what we humans will do with the world if there is no superintelligence. I don't think it's as if the world is in a safe pair of hands. if it remains under human control. And there are all kinds of different ways in which that could go wrong. And if you imagine sort of appointing some randomly selected human and then upgrading their intelligence to some radically superintelligent level, whilst l assume they retained their sort of their values. Like that would also be pretty scary. In fact, maybe more scary than what we would get with a human with an AI That reached superintelligence. And I would go further, like even if you imagine actually carefully selecting a human for this role, like you picked a really nice person, is really kind and friendly and anchored, like you'd still I think have to worry to some extent what would happen when the context for that human changes so radically that they are super intelligent and in control of the world and just see new considerations that we've never dreamt of. You know, maybe maybe they would remain like something that would actually be beneficial to the world, but just how confident can we be that like I mean even if even if it were me myself, like if I were suddenly like uplifted to some super intelligence, who knows really what I would be doing or wanting in that? I mean, I hope that I would remain like nice, but I there's a limit to how confident we could realistically be about that. So so if we're ever gonna like advance to the next level, that I think there is like an intrinsic, maybe ineliminable degree in to which that will require a leap of faith, like some risk. We don't we can't fully s understand the beyond until we get there. And I think that's true whether we do it with the AI path or with some sort of human human upload enhancement, advancement path.

Roman Yampolskiy, PhD: Some time ago we published a paper on whom to upload first and we basically came to exactly the same conclusions. So this is excellent. I'm very, very happy you said what you said. you mentioned different situations with human brain, split brain, all sorts of variants. somewhat random question. what do you think about acquired savant syndrome?

Nick Bostrom: so i i i w what what is that? Somebody develops at some point in their life some surprising savant capability.

Roman Yampolskiy, PhD: So some are savants from birth, some experience an injury, maybe a drug overdose, and as a result they wake up with skills they never had before. They're now playing piano, doing advanced mathematics, maybe speak Chinese. There is about fifty such cases, well documented, not well explained. I'm really curious if you had any i if you don't, obviously that was not part of anything I shared.

Nick Bostrom: Yeah, I don't I don't know really much. I haven't looked into that. It it would be kind of surprising. I mean I could see some things, like maybe some what would be less surprising is if the capacity was of the form, the ability to paint a really super realistic painting. Because maybe what makes that hard for people is partly that there is like higher levels of visual processing that makes us see the objects and the relationships as opposed to sort of see the pixels in our visual field. And maybe if you sort of knocked out that higher interpretation process, you you would more be direct contact with the sort of the the basic components of the visual scenery. And maybe that would allow you to paint more realistically. That that would be less surprising. Like if somebody who's never practiced music suddenly becomes like an expert pianist, that I that that would surprise me greatly. I would have some prior against that. I haven't looked at the cases, but you know

Roman Yampolskiy, PhD: Could be a fun project to to look at. Yeah, I'm trying to find time to

Nick Bostrom: Yeah, yeah, yeah. I mean y you I I would be then almost tempted to look for sort of well you know, some some weird explanation where they did were they're trained to be like playing the piano as a child and then have repressed the memory or is there like are these fake? I d I don't know. But it it's pointless w without actually having looked at the the cases. I haven't really responsible. Yeah, yeah.

Roman Yampolskiy, PhD: I was gambling with that question completely. I figured you're a genius, maybe you have some acquired cosmic knowledge in that space. Your recent papers, super interesting, but also very unorthodox in the style of writing. More like lists of great ideas. Is that deliberate? Are you hoping to use AI to convert them to academic papers? What is the format for?

Nick Bostrom: well it's partly Because it was a faster way for me to do the writing up stage than sort of developing into a fully formed prose paper with the introduction section and the rest of the apparatus. and then partly an experiment in that I myself I think often would prefer to read a document that had this format of a structured set of bullets. Points with concise claims and then subwallets that explain the parts where the structure of the argument is more immediately visually visible, rather than having to wade through some padded sentences and flowery paragraphs that at least for certain types of material. All of that seems to add effort to the author's job and then add more effort on the reader's job to sort of. extract back out the logical skeleton. And so I thought maybe I'll just as an experiment just try to do the skeleton bit and stop there. So it's kind of more immediately perceptible what what what the argument is and its joints. Now I don't know. I mean whether that's successful or not, I don't know. Like I guess it depends on the taste. And I think there are certain types of writing that would benefit from having the full prose where it it's more like immersing the reader in a certain worldview or perspective and bringing them along and aligning their sort of all the different parts of their reading minds, experiences and emotions that then I think you need to do the fully fledged out version. But if it's more just a logical argument kind of, then you know maybe this more bare bones approach might serve.

Roman Yampolskiy, PhD: D do you use AI in your work as an assistant? Is that something you can just ask AI to finish as a proper academic paper?

Nick Bostrom: I'm I'm sort of pernickety with respect to the exact way that my claims are worded. And so I think I would fret over every single sentence if I had the AI right and I would have to sort of rewrite it in order to get it precisely the way I would feel comfortable saying it. I do use AI, I that's like kind of as a re I mean it's basically like having a team of really bright research assistants I'm called twenty four seven, right? Except Instead of waiting for a week to get the report back, it takes five minutes. So it's like very useful in that respect. and I think increasingly also you can experiment it with it as a sort of brainstorming partner. Like if you have some rough idea, you're not really sure how to sort of get started on developing it or like where you go from the like just having the AI kind of critique it or ask some questions about it, then in forcing you to answer those questions, it can sort of help pull things out of you that might Like catalyse your own thinking.

Roman Yampolskiy, PhD: So overall you say you're more productive now because of AIS? are you also more productive because you now no longer doing all the formal professory things we have to do, committees and such?

Nick Bostrom: yeah, well that that's a relief. Yeah. I mean it's a a lot of different aspects. I'm also like I might might look like less I'm less productive in that there are fewer papers coming out, but I'm also focusing less on writing papers, like academic papers, partly because I think the timelines are shorter now. And so the fifteen years ago you could write a paper, wait for a year or two for it to be published in some academic journal, and then eventually maybe you thought somebody would read it who would then write another paper, and then somebody would do that and have some useful idea. eventually it would trickle out and filter into sort of the global mind and we become more cap now now I think you want to look at ways to have a more direct line towards having an impact. So that you know means maybe more direct conversations with people who are doing things or shorter pieces. I I wouldn't necessarily want to set out on a six year book writing project.

Roman Yampolskiy, PhD: Good. So i it would be harder to publish a list in a research journal, but do you see more citations, more engagement with those new format papers?

Nick Bostrom: yeah, I d I don't really care so much about citations and stuff anymore. And I'm not I it's to some extent I think the action has also moved out of academia. especially with things related to AI. and the center of gravity now is very much well, partly in the actual AI frontier labs and then in in a bunch of other organizations sprung up around it, some nonprofit research organizations and some governments and And and and kind of the the all all the ADI pilled people have left academia and it's moving too slow to really be a big part of it. I think I think there are like other fields it is I think different and I think there are some exceptions. I think one area where academia still can play a useful role is in the ethics of digital minds, where I think it would be useful To have some academic groups looking at that in addition to people in various nonprofits and AI labs. Because I just think it's a kind of topic that benefits from having some anchoring in a more traditional and permanently established location, like universities. It might make it less likely that the whole field suddenly gets sort of excised and tabooed and discarded. if if there are sort of you know professors who are like writing research papers, it's harder to dismiss the whole thing as just kind of nonsense. but yeah, so it it it it's it's a yeah there there there are many ways in which no longer being in a university and particularly being responsible for administrative aspects of running a research institute is is a big relief.

Roman Yampolskiy, PhD: Would you as a general advice recommend people leave academia, asking for friend here?

Nick Bostrom: No, I th I think that depends very much on your personal circumstances. like how onerous is your academic job. If it's a sinecure where you basically get paid for doing what you want to do anyway with minimal responsibilities, then I don't know. there's no particular pressure to leave it. you know, if that were some other position of great impact with favorable conditions being dangled in front of you, then maybe it would more be leaping towards that rather than leaping away from academia. Now on the other hand, if you're sort of grinding away in the pursuit of an academic career where you hope one day, 10 years from now to get tenure and be able to do what you want, but in the meantime you're producing these useless papers to rack up your citation count or something, then I think yes, that would be a strong case for considering that there might be alternatives now. Particularly if it's at all related to AI, where there is so much opportunity and funding and interest. in the wider world.

Roman Yampolskiy, PhD: Your first book, Superintel not first, but your book Superintelligence was published by a very prestigious academic press. Your book Utopia, I think, was published by something else. Is that also part of this deliberate shift away from standard academic institutions?

Nick Bostrom: well there were various considerations. One was that I wanted to get it out as fast as possible and with the various more high profile presses, it looked like the timeline would be around a year from the time when I had finished the manuscript to when it could appear in print. Like maybe you could shape that down to ten months. Some some took even like proposed even longer. whereas with this idea press, which i i is more a sort of bespoke author service where they provide the particular services you pay for them, like printing, proofreading, if you want cover design, etcetera, like you could get it turned around much, much faster. And so I thought if there's like a limited number of years remaining until the singularity, during which my thoughts might still have some benefit or relevance, I I didn't want to sort of burn up twenty percent of the remaining time on just waiting for the publisher to Like actually print out the physical copies. also it gives you more control in terms of things like title and cover design and stuff like that. You might might think that it would have been better to have some publisher doing signing the cover, given that what ended up is maybe something that only appeals to some. But nevertheless, that was another consideration.

Roman Yampolskiy, PhD: And between the two books, if we had a debate between Nick Bostrom twenty fourteen, Nick Bostrom twenty twenty four, what would be the main points of disagreement? What who would win?

Nick Bostrom: I don't know. I mean I I like to think that they would get along. I I think my fundamental view hasn't really changed. I think the world has changed, so the specific things that I think are useful to say at the moment are different. And also I already said some of the other things, so I don't want to just keep saying the same things that have already been said. but all along I thought There are two sides to this coin. There's like the the the the risk side and then the upside, like if things go well, it's hugely exciting and they're always been there. But in any particular work, it often makes sense to define the scope and try to really do a good job on a more narrow question rather than try to answer all questions in one go, because that usually just results in a sort of superficial treatment. doesn't really advance anything. And this is, I think, a common source of misunderstanding about my position is that you get various people who have read one thing that I've written, and depending on which thing that is, they then have some weird caricature of what they actually believe. Because they sort of maybe maybe the particular paper says that if you accept these assumptions, then these consequences follow. And then they think I'm the person who advocates for these consequences. Whereas maybe all that paper was doing is examine what happens if you accept these assumptions. Then maybe another paper looks at the question: what if you reject those assumptions? What follows them? And then the idea is that you make incremental progress by sort of breaking out sub-questions and gradually gaining a little bit illumination of the various parts. About if you try to put all of that together into some overall worldview, then it would be much more I would like to think a less crazy and less fanatical than some people might assume if they've just seen like one side. I I think I'm I'm looking I have we have to maybe wrap.

Roman Yampolskiy, PhD: Absolutely. Let me ask you one final question. You have to come up with a clickbait title for this interview. What is it going to be?

Nick Bostrom: that's that's more like your jobs, right? You you

Roman Yampolskiy, PhD: I figured you you may have some ideas about how you want your views to be presented.

Nick Bostrom: well I yeah, yeah. I mean

Roman Yampolskiy, PhD: No pressure, of course. I'm joking. We'll use AI to do it, don't worry about it. But I'm still curious if you have an idea.

Nick Bostrom: well I mean clickbaited titles often don't bear much resemblance to what's actually inside the video. so I guess something like I don't know, like Bostrom's tips for how to lose weight and make a million dollars and become a better lover would probably get a lot of clicks. I don't for the record,

Roman Yampolskiy, PhD: I will run it as one of the

Nick Bostrom: I don't recommend Choosing that one.

Roman Yampolskiy, PhD: YouTube allows me to test multiple options. It will be one of them, I promise.

Nick Bostrom: Well, some things should not even be put to the test.

Roman Yampolskiy, PhD: Thank you so much for your time, for your brilliance, for your direction for humanity. I hope we have more time than your publishing deadlines would suggest. You worried about losing one year before Singularity. I have hope that we have a little more left, but I would love to see you in a few years to see how it turned out.

Nick Bostrom: Yeah, yeah, well and thank you, Roman. good to talk to you.