Astral Codex Ten

Bostrom

15 min read

Bostrom

Article

Bostrom is a recurring person in the Astral Codex Ten archive, appearing 9 times across 9 issues between July 30, 2021 and October 17, 2024. The archive places it in contexts such as ""in 2012-2014, Muller and Bostrom surveyed 550 people""; “If they’re still working on alignment (which would be profoundly weird for many reasons) I expect them to talk about Bostrom”; “alien AIs frequently namedrop Bostrom”. It most often appears alongside Nick Bostrom, Eliezer Yudkowsky, China.

Metadata

  • Category: People
  • Mention count: 9
  • Issue count: 9
  • First seen: July 30, 2021
  • Last seen: October 17, 2024

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Updated Look At Long-Term AI Risks
July 30, 2021 · Original source
1. Superintelligence: This is the "classic" scenario that started the field, ably described by people like Nick Bostrom and Eliezer Yudkowsky. AI progress goes from human-level to vastly-above-human-level very quickly, maybe because slightly-above-human-level AIs themselves are speeding it along, or maybe because it turns out that if you can make an IQ 100 AI for $10,000 worth of compute, you can make an IQ 500 AI for $50,000. You end up with one (or a few) completely unexpected superintelligent AIs, which wield far-future technology and use it in unpredictable ways based on untested goal structures.
Starting from the beginning: in 2012-2014, Muller and Bostrom surveyed 550 people with various levels of claim to the title "AI expert" on the future of AI. People in philosophy of AI or other very speculative fields gave numbers around 20% chance of AI causing an "existential catastrophe" (eg human extinction); people in normal technical AI research gave numbers around 7%. In 2016-2017, Grace et al surveyed 1634 experts, 5% of whom predicted an extremely catastrophic outcome. Both of these surveys were vulnerable to response bias (eg the least speculative-minded people might think the whole issue was stupid and not even return the survey).
Inline links: Muller and Bostrom, Grace et al
Contra Hoel On Aristocratic Tutoring
March 22, 2022 · Original source
If people are still working on AI a hundred years from now, I expect them to talk about Hinton in the same way biologists talk about Darwin now. If they’re still working on alignment (which would be profoundly weird for many reasons) I expect them to talk about Bostrom and various other people I won’t name because some of them read this blog and don’t need bigger egos.
Janus' Simulators
January 26, 2023 · Original source
In the early 2000s, the early AI alignment pioneers - Eliezer Yudkowsky, Nick Bostrom, etc - deliberately started the field in the absence of AIs worth aligning. After powerful AIs existed and needed aligning, it might be too late. But they could glean some basic principles through armchair speculation and give their successors a vital head start.
Just as hucksters frequently namedrop Jesus so their marks think they’re good Christians so alien AIs frequently namedrop Bostrom so their marks think they’re aligned. Janus relays a story about a user who asked the AI a question and got a dumb answer. When the user re-prompted GPT with “how would a super-smart AI answer this question?” it gave him a smart answer. Why? Because it wasn’t even trying to answer the question the first time - it was trying to complete a text about the question. The second time, the user asked it to complete a text about a smart AI answering the question, so it gave a smarter answer.
Inline links: relays a story about
Bostrom’s Superintellence tried to argue that oracles were less safe than they might naively appear. Some oracles might be kind of like agents whose goal is to answer questions. And agents are inherently dangerous. What if it tried to take over the world to get more compute to answer questions better? What if it reduced the universe to a uniform goo, so that it could answer every question with “a uniform goo” and be right? There were lots of scenarios like these; I could never tell whether or not they were too silly to take seriously.
Pause For Thought: The AI Pause Debate
October 05, 2023 · Original source
HOW LONG TO PAUSE. The biggest disadvantage of pausing for a long time is that it gives bad actors (eg China)1 a chance to catch up. Suppose the West is right on the verge of creating dangerous AI, and China is two years away. It seems like the right length of pause is 1.9999 years, so that we get the benefit of maximum extra alignment research and social prep time, but the West still beats China. Obviously the problem with the Surgical Pause is that we might not know when we’re on the verge of dangerous AI, and we might not know how much of a lead “the good guys” have. Surgical Pause proponents suggest being very conservative with both free variables. This is less of a well-thought-out plan and more saying “come on guys, let’s at least try to be strategic here”. At the limit, it suggests we probably shouldn’t pause for six months, starting right now. Since this involves leading labs burning their lead time for safety, in theory it could be done unilaterally by the single leading lab, without international, governmental, or even inter-lab coordination. But you could buy more time if you got those things too. Some leading labs have promised to do this when the time is right - for example OpenAI and (a previous iteration of) DeepMind - with varying levels of believability. AnonResearcherAtMajorAILab discussed some of the strategy here in Aim For Conditional AI Pauses, and this Less Wrong post is also very good. Regulatory Pause: If one benefit of the Simple Pause is to use the time to prepare for AI socially and politically, maybe we should just pause until we’ve completed social and political preparations. David Manheim suggests a monitoring agency like the FDA. It would “fast-track” small AIs and trivial re-applications of existing AIs, but carefully monitor new “frontier models” for signs of danger. Regulators might look for dangerous capabilities by asking AIs to hack computers or spread copies of themselves, or test whether they’ve been programmed against bias/misinformation/etc. We could pause only until we’ve set up the regulatory agency, and take hostile actions (like restrict chip exports) only to other countries that don’t cooperate with our regulators or set up domestic regulators of their own. Many people in tech are regulation-skeptical libertarians, but proponents point out that regulation fails in a predictable direction: it usually does successfully prevent bad things, it just also prevents good things too. Since the creation of the Nuclear Regulatory Commission in 1975, there has never been a major nuclear accident in the US. And sure, this is because the NRC prevented any nuclear plants from being built in the United States at all from 1975 to 2023 (one was finally built in July). Still, they technically achieved their mandate. Likewise, most medications in the US are safe and relatively effective, at the cost of an FDA approval process being so expensive that we only get a tiny trickle of new medications each year and hundreds of thousands of people die from unnecessary delays. But medications are safe and effective. Or: San Francisco housing regulators almost never approve new housing, so housing costs millions of dollars and thousands of San Franciscans are homeless - but certainly there’s no epidemic of bad houses getting approved and then ruining someone’s view or something. If we extrapolate this track record to AI, AI regulators will be overcautious, progress will slow by orders of magnitude or stop completely - but AIs will be safe. This is a depressing prospect if you think the problems from advanced AI would be limited to more spam or something. But if you worry about AI destroying the world, maybe you should accept a San-Francisco-housing-level of impediment and frustration. A regulatory pause could be better than a total stop if you think it will be more stable (lots of industries stay heavily regulated forever, and only a few libertarians complain), or if you think maybe the regulator will occasionally let a tiny amount of safe AI progress happen. But it could be worse than a total stop if you expect continued progress will eventually produce unsafe AIs regardless of regulation. You might expect this if you’re worried about deceptive alignment, eg superintelligent AIs that deliberately trick regulators into thinking they’re safe. Or you might think AIs will eventually be so powerful that they can endanger humanity from a walled-off test environment even before official approval. The classic Bostrom/Yudkowsky model of alignment implies both of these things. David Manheim and Thomas Larsen set out their preferred versions of this strategy in What’s In A Pause? and Policy Ideas For Mitigating AI Risk. Total Stop: If you expect AIs to exhibit deceptive alignment capable of fooling regulators, or to be so dangerous that even testing them on a regulator’s computer could be apocalyptic, maybe the only option is a total stop. It’s tough to imagine a total stop that works for more than a few years. You have at least three problems: NON-PARTICIPANTS. As with any pause proposal, unfriendly countries (eg China) can keep working on AI. You can refuse to export chips to them, which will slow them down a little, but their own chips will eventually be up to the task. You will either need a diplomatic miracle, or willingness to resort to less diplomatic forms of coercion. This doesn’t have to be immediate war: Israel has come up with “creative” ways to slow Iran’s nuclear program, and countries trying to frustrate China’s chip industry could do the same. But great powers playing these kinds of games against each other risks wider conflict.
Inline links: 1, OpenAI, DeepMind, Aim For Conditional AI Pauses, this Less Wrong post, look for dangerous capabilities, finally built, What’s In A Pause?, Policy Ideas For Mitigating AI Risk
Nora thought it was important to give alignment researchers advanced models to experiment with, because the sort of armchair alignment research before interesting AIs existed (eg Bostrom’s Superintelligence) wasn’t just wrong, but fostered dead-end worse-than-nothing paradigms that continue to confuse the field. Daniel Filan objected that Bostrom got some things right and even described something like the direction that modern alignment research is taking. There was a long argument about this, which I think reduces to “Bostrom said some useful theoretical things, speculated about practical direction, and a few of his speculations were right but most now seem outdated”.
Inline links: objected
Links For November 2023
December 01, 2023 · Original source
35: The accelerationists (and Tyler Cowen) are trying to trick people into thinking Nick Bostrom “regrets focusing on AI risk”. Please read the actual interview, where Bostrom says:
Inline links: The accelerationists, Tyler Cowen, the actual interview
…while saying that we also need to make sure we don’t overshoot, never develop AI at all, and stagnate forever. This is my position too (see eg the second part of Part III here). I think it’s a common position! But the idea that Bostrom has “recanted” his concern with AI risk is false. Please apply a Gell-Mann amnesia correction in anything else you read about AI from anyone who said this.
Inline links: here
I also think the rest of the Bostrom interview is great and (unsurprisingly) models a really thoughtful way of balancing these risks.
Profile: The Far Out Initiative
May 15, 2024 · Original source
Turn-of-the-21st-century Oxford was an exciting place. Derek Parfit was leading a renaissance in utilitarian thought. New technologies like the personal computer, the Internet, and the Human Genome Project were inspiring a new generation of transhumanists. Out of this milieu, philosophers like Nick Bostrom, Will MacAskill, and Toby Ord were laying the groundwork for what would become the rationalist and effective altruist movements. Utilitarians, they argued, were charged with relieving the suffering of the world as quickly and effectively as possible. Technology offered new opportunities to do this at scale. This could be ending poverty and curing diseases (if you were well-grounded in the present moment) or creating a superintelligence to lead us to a post-scarcity future (if you were feeling more ambitious).
Pearce’s old friend Nick Bostrom imagines a future of superintelligence. But Pearce will count his own contribution complete if he gives us superhappiness, supermeaning, superbeauty, and superspirituality. And why shouldn’t he? People on LSD and MDMA have all of these things. All we need to do is figure out how to do it without the trippy hallucinations, urge to go to raves, and occasional neurotoxicity. Don’t say it’s impossible! All you need to do is find the right Scottish person!
Inline links: and occasional neurotoxicity
This is Pearce’s thesis. It’s not as popular as the normal effective altruism that just tries to help solve poverty and cure diseases. While Ord and Bostrom and MacAskill got followers and press coverage and friendly billionaires, Pearce and his movement (“suffering abolitionism”) got a few very devoted email correspondents.
Interview Day At Thiel Capital
September 03, 2024 · Original source
“Bostrom’s original simulation argument said that if future generations simulated the past, there would be far more of these simulations than there were actual pasts. He thought maybe people would simulate the past to learn about the branching pattern of history. That’s the kind of mistake only a philosopher could make. If we look at existing media consumption - whether it’s videos, RPGs, or incipient VR properties - by far the most common category is porn. Even if you limit your search to historical media, there are a hundred bodice-rippers for every sober investigation of Victorian lifestyles.”
Sakana, Strawberry, and Scary AI
September 18, 2024 · Original source
But nobody finds this scary. Nobody thinks “oh, yeah, Bostrom and Yudkowsky were right, this is that AI safety thing”. It’s just another problem for the cybersecurity people. Sometimes Excel inappropriately converts things to dates; sometimes GPT-6 tries to upload itself into an F-16 and bomb stuff. That specific example might be kind of a joke. But thirty years ago, it also would have sounded pretty funny to speculate about a time when “everyone knows” AIs can write poetry and develop novel mathematics and beat humans at chess, yet nobody thinks they’re intelligent.
Book Review: Deep Utopia
October 17, 2024 · Original source
If wireheading seems too meaningless, you can add in wireheaded-meaning. People often say that an MDMA trip or mystical vision was the most meaningful experience of their lives. It would be trivial for our Deep Utopians to hack your brain to see a world in a grain of sand or heaven in a wildflower. We're gonna mean so much, you might even get tired of meaning. And you'll say 'please, please, it's too much meaning. We can't take it anymore, Professor Bostrom, it's too much!'
Oxford philosopher Nick Bostrom got famous for asking “What if technology is really really bad?” He helped define ‘existential risk’, popularize fears of malevolent superintelligence, and argue that we were living in a ‘vulnerable world’ prone to physical or biological catastrophe.
If you made Zizek write fiction, you would get Deep Utopia. The book takes the form of a story. The story is: some young people go to a lecture series by Nick Bostrom. At the lecture, Bostrom says [commence 468 pages of Bostrom describing his theory of purpose in utopia]. Then the young people go to a party, then go home. The end.