Astral Codex Ten

Nick Bostrom

14 min read

Nick Bostrom

Article

Nick Bostrom is a recurring person in the Astral Codex Ten archive, appearing 13 times across 13 issues between July 30, 2021 and October 17, 2024. The archive places it in contexts such as ""described by people like Nick Bostrom and Eliezer Yudkowsky""; “described by people like Nick Bostrom”; “Eliezer Yudkowsky, Robin Hanson, Nick Bostrom, and other intellectuals”. It most often appears alongside Bostrom, Eliezer Yudkowsky, Toby Ord.

Metadata

  • Category: People
  • Mention count: 13
  • Issue count: 13
  • First seen: July 30, 2021
  • Last seen: October 17, 2024

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Updated Look At Long-Term AI Risks
July 30, 2021 · Original source
1. Superintelligence: This is the "classic" scenario that started the field, ably described by people like Nick Bostrom and Eliezer Yudkowsky. AI progress goes from human-level to vastly-above-human-level very quickly, maybe because slightly-above-human-level AIs themselves are speeding it along, or maybe because it turns out that if you can make an IQ 100 AI for $10,000 worth of compute, you can make an IQ 500 AI for $50,000. You end up with one (or a few) completely unexpected superintelligent AIs, which wield far-future technology and use it in unpredictable ways based on untested goal structures.
Starting from the beginning: in 2012-2014, Muller and Bostrom surveyed 550 people with various levels of claim to the title "AI expert" on the future of AI. People in philosophy of AI or other very speculative fields gave numbers around 20% chance of AI causing an "existential catastrophe" (eg human extinction); people in normal technical AI research gave numbers around 7%. In 2016-2017, Grace et al surveyed 1634 experts, 5% of whom predicted an extremely catastrophic outcome. Both of these surveys were vulnerable to response bias (eg the least speculative-minded people might think the whole issue was stupid and not even return the survey).
Inline links: Muller and Bostrom, Grace et al
Why Do I Suck?
February 02, 2022 · Original source
It still is! But in the same sense that I was clearing a personal backlog of unwritten-up ideas, the rationalist community was clearing a backlog of scientific and philosophical ideas sitting in journals or obscure old books that it turned out were really interesting to a lot of people. The early Internet provided a critical mass where people interested in cognition and math and the future could suddenly all share the parts of the puzzle they knew about with each other and make rapid progress. Eliezer Yudkowsky, Robin Hanson, Nick Bostrom, and other intellectuals all had their own backlog of stuff which had probably been published in journals or something but which the wider world had yet to appreciate. I was the biggest-name blogger who was sitting around listening to them talk about it, so I got access to a stream of amazing content that most people didn’t know about.
Open Thread 217
March 27, 2022 · Original source
2: The effective altruists I know are really excited about Carrick Flynn for Congress (he’s running as a Democrat in Oregon). Carrick has fought poverty in Africa, worked on biosecurity and pandemic prevention since 2015, and is a world expert on the intersection of AI safety and public policy (see eg this paper he co-wrote with Nick Bostrom). He also supports normal Democratic priorities like the environment, abortion rights, and universal health care (see here for longer list). See also this endorsement from biosecurity grantmaker Andrew SB.
Inline links: Carrick Flynn for Congress, this paper, here, this endorsement
Open Thread 239
August 28, 2022 · Original source
2: The ACX podcast team is involved in a new project, Radio Bostrom, broadcasting works by futurist philosopher Nick Bostrom.
Inline links: Radio Bostrom
How Do AIs' Political Opinions Change As They Get Smarter And Better-Trained?
January 03, 2023 · Original source
You know all that stuff that Nick Bostrom and Eliezer Yudowsky and Stuart Russell have been warning us about for years, where AIs will start seeking power and resisting human commands? I regret to inform you that if you ask AIs whether they will do that stuff, they say yeah, definitely.
Janus' Simulators
January 26, 2023 · Original source
In the early 2000s, the early AI alignment pioneers - Eliezer Yudkowsky, Nick Bostrom, etc - deliberately started the field in the absence of AIs worth aligning. After powerful AIs existed and needed aligning, it might be too late. But they could glean some basic principles through armchair speculation and give their successors a vital head start.
Just as hucksters frequently namedrop Jesus so their marks think they’re good Christians so alien AIs frequently namedrop Bostrom so their marks think they’re aligned. Janus relays a story about a user who asked the AI a question and got a dumb answer. When the user re-prompted GPT with “how would a super-smart AI answer this question?” it gave him a smart answer. Why? Because it wasn’t even trying to answer the question the first time - it was trying to complete a text about the question. The second time, the user asked it to complete a text about a smart AI answering the question, so it gave a smarter answer.
Inline links: relays a story about
Bostrom’s Superintellence tried to argue that oracles were less safe than they might naively appear. Some oracles might be kind of like agents whose goal is to answer questions. And agents are inherently dangerous. What if it tried to take over the world to get more compute to answer questions better? What if it reduced the universe to a uniform goo, so that it could answer every question with “a uniform goo” and be right? There were lots of scenarios like these; I could never tell whether or not they were too silly to take seriously.
Crowds Are Wise (And One's A Crowd)
February 06, 2023 · Original source
This looks like some specific elegant curve, but which one? A real statistician would be able to give a good answer to this question. I can’t, but after mashing some buttons on my statistics program and seeing what happened, I got the equation -Epistemic status: Wild speculation outside the limits of my competence- 1/ERROR = 2.34 + [1.8 * ln(CROWD_SIZE)]…which does okay at predicting the n=100 data point too. This equation implies that as crowd size approaches infinity, error approaches zero (albeit very slowly). But I included that assumption when choosing the equation - I didn’t test it. You can also imagine that there’s some consistent bias. For example, if the most commonly used map projection is distorted such that eyeballing the distance on a map perfectly would leave you off by 100 km, an infinitely-sized crowd might converge to an error of 100 km. I can’t tell if that’s going on here or not. For what it’s worth, taking the equation seriously suggests that if all 8 billion people on Earth took my survey, we would have gotten within 50 km of the true distance. Nick Bostrom speculates that in the far future, a multigalactic supercivilization might be able to support 10^46 simulated humans per century. If all of them took my survey, we could get within 12 km. Can You Really Do Wisdom Of Crowds With Yourself? As mentioned above, the average respondent was off by 918 km on their first guess. They were off by 967 km on their second guess. And on the average of their guesses, they were off by . . . it depends if you mean arithmetic or geometric average. The arithmetic average was better, 916 km. The geometric average was worse, 940 km. Arithmetic average is more commonly used. But I’d been using geometric average before, to deal with outliers. But this is a simple averaging of two quantities, where “outlier” is meaningless. So maybe arithmetic mean is more appropriate again? If we remove all ridiculous outliers from the data (anything above 40000 km, which would get you all the way around the Earth, or below 200 km, which wouldn’t even get you out of France) the picture is similar. Error on the first guess goes down to 858 km, on the second to 898 km, on the geometric mean to 873 km, and on the arithmetic mean to 845 km. Now all differences are significant at p < 0.001. Notice that two guesses from the same person were much less effective than two guesses from two different people, bringing the error down by 2 - 13 km instead of 200. This analysis is limited by having only one question, meaning that I can’t test whether the choices I made were good vs. p-hacking. If I had another question like this, I would like to confirm that removing outliers and using arithmetic instead of geometric mean for the stage where you average the two guesses still produces better results. At this point I can just say that I’ve found suggestive evidence that the wisdom-of-crowds-with-yourself hypothesis holds. Is the bound as number of guesses goes to infinity still zero? Can you get any question right just by guessing thousands of times, then averaging the results? Surely the answer has to be “no” - otherwise it would be too OP. Van Dolder, Van Den Assem Van Dolder and Van Den Assem did a much bigger wisdom-of-inner-crowds experiment, published here in Nature Human Behavior. It answers the “infinite inner crowd” question and tells us more about how the phenomenon works. VD and VDA got data from a Dutch casino that had a “guess the number of objects in a glass container” contest each year for several years (the real number was usually in the tens of thousands). Several hundred thousand people played, some more than once. Here are their results: If I’m reading this right, they find: Both inner and outer (ie real) crowds get more accurate as crowd size increases.
Inline links: speculates that, here, https://substackcdn.com/image/fetch/$s_!6WSb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2db186c6-b785-4513-bb9a-7f971afc817f_860x773.png
The Extinction Tournament
July 20, 2023 · Original source
Question 51 asks when we will have AGI (the resolution criteria are that whatever Nick Bostrom says goes).
Everyone agrees it’s pretty likely we’ll have AGI (as per Bostrom) by 2100, although the domain experts are a little more convinced than the superforecasters.
Contra DeBoer On Movement Shell Games
November 30, 2023 · Original source
EA might have screwed this up worse than some other groups, but I don’t think a movement our size is capable of rebranding. We just have to eat the loss. If we were optimizing entirely for clarity and not for attractive-soundingness, I’d go for Systematic Altruism on the one side, and The Network Of People Who All Pursue Systematic Altruism Together In A Way Causally Downstream Of Toby Ord, Will MacAskill, And Nick Bostrom (TONOPWAPSATIAWCDOTOWMAANB) on the other.
Links For November 2023
December 01, 2023 · Original source
35: The accelerationists (and Tyler Cowen) are trying to trick people into thinking Nick Bostrom “regrets focusing on AI risk”. Please read the actual interview, where Bostrom says:
Inline links: The accelerationists, Tyler Cowen, the actual interview
…while saying that we also need to make sure we don’t overshoot, never develop AI at all, and stagnate forever. This is my position too (see eg the second part of Part III here). I think it’s a common position! But the idea that Bostrom has “recanted” his concern with AI risk is false. Please apply a Gell-Mann amnesia correction in anything else you read about AI from anyone who said this.
Inline links: here
I also think the rest of the Bostrom interview is great and (unsurprisingly) models a really thoughtful way of balancing these risks.
Profile: The Far Out Initiative
May 15, 2024 · Original source
Turn-of-the-21st-century Oxford was an exciting place. Derek Parfit was leading a renaissance in utilitarian thought. New technologies like the personal computer, the Internet, and the Human Genome Project were inspiring a new generation of transhumanists. Out of this milieu, philosophers like Nick Bostrom, Will MacAskill, and Toby Ord were laying the groundwork for what would become the rationalist and effective altruist movements. Utilitarians, they argued, were charged with relieving the suffering of the world as quickly and effectively as possible. Technology offered new opportunities to do this at scale. This could be ending poverty and curing diseases (if you were well-grounded in the present moment) or creating a superintelligence to lead us to a post-scarcity future (if you were feeling more ambitious).
Pearce’s old friend Nick Bostrom imagines a future of superintelligence. But Pearce will count his own contribution complete if he gives us superhappiness, supermeaning, superbeauty, and superspirituality. And why shouldn’t he? People on LSD and MDMA have all of these things. All we need to do is figure out how to do it without the trippy hallucinations, urge to go to raves, and occasional neurotoxicity. Don’t say it’s impossible! All you need to do is find the right Scottish person!
Inline links: and occasional neurotoxicity
This is Pearce’s thesis. It’s not as popular as the normal effective altruism that just tries to help solve poverty and cure diseases. While Ord and Bostrom and MacAskill got followers and press coverage and friendly billionaires, Pearce and his movement (“suffering abolitionism”) got a few very devoted email correspondents.
Contra DeBoer On Temporal Copernicanism
September 10, 2024 · Original source
The world’s leading expert on anthropic reasoning is probably Oxford philosophy professor Nick Bostrom, who literally wrote the book on the subject. Awkwardly for Freddie, Bostrom is also one of the founders of the modern singularity movement. This is because, understood correctly, anthropics provides no argument against a singularity or any other transhumanist idea, and might (weakly) support them.
Inline links: literally wrote the book
Book Review: Deep Utopia
October 17, 2024 · Original source
Oxford philosopher Nick Bostrom got famous for asking “What if technology is really really bad?” He helped define ‘existential risk’, popularize fears of malevolent superintelligence, and argue that we were living in a ‘vulnerable world’ prone to physical or biological catastrophe.
If you made Zizek write fiction, you would get Deep Utopia. The book takes the form of a story. The story is: some young people go to a lecture series by Nick Bostrom. At the lecture, Bostrom says [commence 468 pages of Bostrom describing his theory of purpose in utopia]. Then the young people go to a party, then go home. The end.
When there are little side boxes going into more detail about a certain topic, they’re described as “handouts” at the “lecture”. Occasionally, not-especially-wacky things happen, like the fire alarm goes on, or the lights go out. Sometimes when Bostrom wants to play devil’s advocate, he puts his question in the mouth of one of the students. Otherwise, it’s pretty much what you would expect from a 468-page dense philosophy tome with a fig leaf of “And some students went to a lecture where Nick Bostrom said…” at the beginning.