Astral Codex Ten

Toby Ord

16 min read

Toby Ord

Article

Toby Ord is a recurring person in the Astral Codex Ten archive, appearing 9 times across 9 issues between January 29, 2021 and February 06, 2025. The archive places it in contexts such as “People like Toby Ord tried to calculate the risk of every kind of disaster”; “Toby Ord (here standing in for the broader existential-risk-quantifying community) has estimated”; “Effective altruism is now a semi-organized movement, with leaders like Will MacAskill and Toby Ord”. It most often appears alongside Nick Bostrom, 80,000 Hours, EA.

Metadata

  • Category: People
  • Mention count: 9
  • Issue count: 9
  • First seen: January 29, 2021
  • Last seen: February 06, 2025

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Contra Weyl On Technocracy
January 29, 2021 · Original source
Weyl wrote this essay a few months before COVID, so his pooh-poohing of the idea that there might be a biological catastrophe is an unfortunate anachronism. But I think it's important to note that we got this right (and he got it wrong) precisely because we "privilege rationalist approaches over all other forms of knowledge-making". People like Toby Ord tried to calculate the risk of every kind of disaster and how bad it would be - and at the same time Weyl was making fun of us for caring about biological catastrophes, Ord was writing about how the numbers suggested zoonotic diseases from bats could cause catastrophic pandemics. This kind of work ultimately led to EA flagship group Open Philanthropy Project spending almost $50 million on its Biosecurity And Pandemic Preparedness Program between 2014 and 2019; if other people had taken a few minutes to read our arguments instead of chiding us for how naive it is to prioritize things based on rational methods, maybe the world would have been more prepared.
Inline links: like Toby Ord, Biosecurity And Pandemic Preparedness Program
Why Not Slow AI Progress?
August 08, 2022 · Original source
Toby Ord (here standing in for the broader existential-risk-quantifying community) has estimated the risk of extinction-level asteroid impacts as 0.0001% per century, and the risk of extinction from building AI too fast as 10%. So as written, this argument isn’t very good. But you could revise it to be about metaphorical “asteroids” like superplagues or nuclear war. Altman has also expressed concern about AI causing inequality, for example if rich people use it to replace all labor and reap all the gains for themselves. OpenAI was originally founded as a nonprofit in a way that protected against that, so maybe he thought that made it preferable to DeepMind.
Inline links: has estimated
Effective Altruism As A Tower Of Assumptions
August 24, 2022 · Original source
There’s a lot of commentary. Effective altruism is now a semi-organized movement, with leaders like Will MacAskill and Toby Ord and institutions like the Open Philanthropy Project. It’s produced a vast literature on effective charities, ranging from how to best prevent malaria to how to promote animal welfare to speculative scenarios about AI apocalypse. These aren’t above criticism, and lots of people have criticized them. But if you criticize them successfully, and feel like they’re discredited, then you’re back at the basic tenets of the movement again.
The Extinction Tournament
July 20, 2023 · Original source
You might notice that all of these numbers are pretty low! I’ve previously said I thought there was a 33% chance of AI extinction alone (and lots of people are higher than me). Existential risk expert Toby Ord estimated a 16% total chance of extinction by 2100, which is 16x higher than these superforecasters and 2.5x higher than these domain experts. In some sense, this is great news. These kinds of expert + superforecasting tournaments seem trustworthy. Should we update our risk of human extinction downward? Cancelling The Apocalypse? It’s weird that there’s so much difference between experts and superforecasters, and awkward for me that both groups are so far away from my own estimates and those of people I trust (like Toby). Is there any reason to doubt the results? Were the incentives bad? The subreddit speculates about this - after all, you can’t get paid, or congratulated, or given a trophy, if the world goes extinct. Does that bias superforecasters - who are used to participating in prediction markets and tournaments - downward? What about domain experts, who might be subconsciously optimizing for prestige and reputation? This tournament tried to control for that in a few ways. First, most of the monetary incentives were for things other than predicting extinction. There were incentives for making good arguments that persuaded other participants, for correctly predicting intermediate steps to extinction (for example, a small pandemic, or a limited nuclear exchange), or for correctly guessing what other people would guess (this technique, called “reciprocal scoring”, has been validated in past experiments). Second, this wasn’t really an incentive-based prediction market. Although they kept a few incentives as described above, it was mostly about asking people who had previously demonstrated good predictive accuracy to give their honest impressions. At some point you just have to trust that, absent incentives either way, reasonable people with good track records can be smart and honest. Third, a lot of the probabilities here were pretty low. For example, the superforecasters got an 0.4% probability of AI-based extinction, compared to the domain experts’ 3%. At these levels it’s probably not worth optimizing your answers super-carefully to get a tiny amount of extra money or credibility. If it’s the year 2100, and we didn’t die from AI, who was right - the people who said there was a 3% chance, or the people who said there was an 0.4% chance? Everyone in this tournament was smart enough to realize that survival in one timeline wouldn’t provide much evidence either way. As tempting as it is to dismiss this surprising result with an appeal to the incentive structure, we’re not going to escape that easily. Were the forecasters stupid? Aside from the implausibility of dozens of top superforecasters and domain experts being dumb, both groups got easy questions right. The bio-risks questions are a good benchmark here: There are centuries’ worth of data on non-genetically-engineered plagues to give us base rates; these give us a base rate of ~25% per century = 20% between now and 2100. But we have better epidemiology and medicine than most of the centuries in our dataset. The experts said 8% chance and the superforecasters said 4% chance, and both of those seem like reasonable interpretations of the historical data to me. The “WHO declares emergency” question is even easier - just look at how often it’s done that in the past and extrapolate forward. Both superforecasters and experts mostly did that. Likewise, lots of scientists have put a lot of work into modeling the climate, there aren’t many surprises there, and everyone basically agreed on the extent of global warming: Wherever there was clear past data, both superforecasters and experts were able to use it correctly and get similar results. It was only when they started talking about things that had never happened before - global nuclear war, bioengineered pandemics, and AI - that they started disagreeing. Were the participants out of their depth? Peter McCluskey, one of the more-AI-concerned superforecasters in the tournament, wrote about his experience on Less Wrong. Quoting liberally: I signed up as a superforecaster. My impression was that I knew as much about AI risk as any of the subject matter experts with whom I interacted (the tournament was divided up so that I was only aware of a small fraction of the 169 participants). I didn't notice anyone with substantial expertise in machine learning. Experts were apparently chosen based on having some sort of respectable publication related to AI, nuclear, climate, or biological catastrophic risks. Those experts were more competent, in one of those fields, than news media pundits or politicians. I.e. they're likely to be more accurate than random guesses. But maybe not by a large margin […] The persuasion seemed to be spread too thinly over 59 questions. In hindsight, I would have preferred to focus on core cruxes, such as when AGI would become dangerous if not aligned, and how suddenly AGI would transition from human levels to superhuman levels. That would have required ignoring the vast majority of those 59 questions during the persuasion stages. But the organizers asked us to focus on at least 15 questions that we were each assigned, and encouraged us to spread our attention to even more of the questions […] Many superforecasters suspected that recent progress in AI was the same kind of hype that led to prior disappointments with AI. I didn't find a way to get them to look closely enough to understand why I disagreed. My main success in that area was with someone who thought there was a big mystery about how an AI could understand causality. I pointed him to Pearl, which led him to imagine that problem might be solvable. But he likely had other similar cruxes which he didn't get around to describing. That left us with large disagreements about whether AI will have a big impact this century. I'm guessing that something like half of that was due to a large disagreement about how powerful AI will be this century. I find it easy to understand how someone who gets their information about AI from news headlines, or from laymen-oriented academic reports, would see a fair steady pattern of AI being overhyped for 75 years, with it always looking like AI was about 30 years in the future. It's unusual for an industry to quickly switch from decades of overstating progress, to underhyping progress. Yet that's what I'm saying has happened. I've been spending enough time on LessWrong that I mostly forgot the existence of smart people who thought recent AI advances were mostly hype. I was unprepared to explain why I thought AI was underhyped in 2022. Today, I can point to evidence that OpenAI is devoting almost as much effort into suppressing abilities (e.g. napalm recipes and privacy violations) as it devotes to making AIs powerful. But in 2022, I had much less evidence that I could reasonably articulate. What I wanted was a way to quantify what fraction of human cognition has been superseded by the most general-purpose AI at any given time. My impression is that that has risen from under 1% a decade ago, to somewhere around 10% in 2022, with a growth rate that looks faster than linear. I've failed so far at translating those impressions into solid evidence. Skeptics pointed to memories of other technologies that had less impact (e.g. on GDP growth) than predicted (the internet). That generates a presumption that the people who predict the biggest effects from a new technology tend to be wrong. > Superforecasters' doubts about AI risk relative to the experts isn't primarily driven by an expectation of another "AI winter" where technical progress slows. ... That said, views on the likelihood of artificial general intelligence (AGI) do seem important: in the postmortem survey, conducted in the months following the tournament, we asked several conditional forecasting questions. The median superforecaster's unconditional forecast of AI-driven extinction by 2100 was 0.38%. When we asked them to forecast again, conditional on AGI coming into existence by 2070, that figure rose to 1%. There was also little or no separation between the groups on the three questions about 2030 performance on AI benchmarks (MATH, Massive Multitask Language Understanding, QuALITY). This suggests that a good deal of the disagreement is over whether measures of progress represent optimization for narrow tasks, versus symptoms of more general intelligence. The “won’t understand causality” and “what if it’s all hype” objections really don’t impress me. Many of the people in this tournament hadn’t really encountered arguments about AI extinction before (potentially including the “AI experts” if they were just eg people who make robot arms or something), and a couple of months of back and forth discussion in the middle of a dozen other questions probably isn’t enough for even a smart person to wrap their brain around the topic. Was this tournament done so long ago that it has been outpaced by recent events? The tournament was conducted in summer 2022. This was before ChatGPT, let alone GPT-4. The conversation around AI noticeably changed pitch after these two releases. Maybe that affected the results? In fact, the participants have already been caught flat-footed on one question: A recent leak suggested that the cost of training GPT-4 was $63 million, which is already higher than the superforecasters’ median estimate of $35 million by 2024 has already been proven incorrect. I don’t know how many petaFLOP-days were involved in GPT-4, but maybe that one is already off also. There was another question on when an AI would pass a Turing Test. The superforecasters guessed 2060, the domain experts 2045. GPT-4 hasn’t quite passed the exact Turing Test described in the study, but it seems very close, so much so that we seem on track to pass it by the 2030s. Once again the experts look better than the superforecasters. So is it possible that we, in 2023, now have so much better insight into AI than the 2022 forecasters that we can throw out their results? We could investigate this by looking at Metaculus, a forecasting site that’s probably comparably advanced to this tournament. They have a question suspiciously similar to XPT’s global catastrophe framing: In summer 2022, the Metaculus estimate was 30%, compared to the XPT superforecasters’ 9% (why the difference? maybe because Metaculus is especially popular with x-risk-pilled rationalists). Since then it’s gone up to 38%. Over the same period, Metaculus estimates of AI catastrophe risk went from 6% to 15%. If the XPT superforecasters’ probabilities rose linearly by the same factor as Metaculus forecasters’, they might be willing to update total global catastrophe risk to 11% and AI catastrophe risk to 5%. But the main thing we’ve updated on since 2022 is that AI might be sooner. But most people in the tournament already agreed we would get AGI by 2100. The main disagreement was over whether it would cause a catastrophe once we got it. You could argue that getting it sooner increases that risk, since we’ll have less time to work on alignment. But I would be surprised if the kind of people saying the risk of AI extinction is 0.4% are thinking about arguments like that. So maybe we shouldn’t expect much change. FRI called back a few XPT forecasters in May 2023 to see if any of them wanted to change their minds, but they mostly didn’t. Overall I don’t think this was just a problem of the incentives being bad or the forecasters being stupid. This is a real, strong disagreement. We may be able to slightly increase their forecast based on recent events, but this would only change the estimate a little. Breaking Down The AI Estimate How did the forecasters arrive at their AI estimate? What were the cruxes between the people who thought AI was very dangerous, and the people who thought it wasn’t? You can think of AI extinction as happening in a series of steps: We get human-level AI by 2100.
Inline links: previously said, Existential risk expert Toby Ord, The subreddit, “reciprocal scoring”, https://substackcdn.com/image/fetch/$s_!nQcg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d508cae-def3-4bc3-888f-be7e16032b5c_815x339.png, https://substackcdn.com/image/fetch/$s_!KJ84!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1f1c4fd-5981-458c-959f-bf9a19ff28da_801x129.png, wrote about his experience, Pearl, https://substackcdn.com/image/fetch/$s_!CfZT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2362d361-ae0a-4e4f-ad97-cbeb1fcbe827_817x351.png, the cost of training GPT-4 was $63 million, Metaculus, a question, https://substackcdn.com/image/fetch/$s_!k5Ep!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09cb5518-f8d1-4a98-8c44-97158857dbd8_772x364.png
23
August 28, 2023 · Original source
3: Existential risk expert Toby Ord responds to FRI’s Existential Risk Persuasion Tournament disagreeing with him.
Inline links: responds to FRI’s Existential Risk Persuasion Tournament disagreeing with him
Contra DeBoer On Movement Shell Games
November 30, 2023 · Original source
EA might have screwed this up worse than some other groups, but I don’t think a movement our size is capable of rebranding. We just have to eat the loss. If we were optimizing entirely for clarity and not for attractive-soundingness, I’d go for Systematic Altruism on the one side, and The Network Of People Who All Pursue Systematic Altruism Together In A Way Causally Downstream Of Toby Ord, Will MacAskill, And Nick Bostrom (TONOPWAPSATIAWCDOTOWMAANB) on the other.
Profile: The Far Out Initiative
May 15, 2024 · Original source
Turn-of-the-21st-century Oxford was an exciting place. Derek Parfit was leading a renaissance in utilitarian thought. New technologies like the personal computer, the Internet, and the Human Genome Project were inspiring a new generation of transhumanists. Out of this milieu, philosophers like Nick Bostrom, Will MacAskill, and Toby Ord were laying the groundwork for what would become the rationalist and effective altruist movements. Utilitarians, they argued, were charged with relieving the suffering of the world as quickly and effectively as possible. Technology offered new opportunities to do this at scale. This could be ending poverty and curing diseases (if you were well-grounded in the present moment) or creating a superintelligence to lead us to a post-scarcity future (if you were feeling more ambitious).
Contra DeBoer On Temporal Copernicanism
September 10, 2024 · Original source
What’s Freddie doing wrong, and how can we do better? The following argument is loosely based on one by Toby Ord. Consider three types of events:
Inline links: on one by Toby Ord
Money Saved By Canceling Programs Does Not Immediately Flow To The Best Possible Alternative
February 06, 2025 · Original source
When studying charities, Toby Ord found that of two randomly chosen charities, one will be (on average) 100x more effective than the other. Government programs aren’t charities, but common-sensically we might expect similar dynamics to apply2, and for an unusually good program (like PEPFAR) to be 100x more efficient than one which is somewhere between average and worst.
Inline links: 2