Astral Codex Ten

AI risk

40 min read

AI risk

Article

AI risk is a recurring concept in the Astral Codex Ten archive, appearing 8 times across 8 issues between August 06, 2021 and October 24, 2025. The archive places it in contexts such as “work done in AI risk”; “weird causes (eg AI risk)”; “things many people have found self-evidently absurd (eg … AI risk)“. It most often appears alongside Eliezer Yudkowsky, China, AI.

Metadata

  • Category: Concepts
  • Mention count: 8
  • Issue count: 8
  • First seen: August 06, 2021
  • Last seen: October 24, 2025

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Highlights From The Comments On Acemoglu And AI
August 06, 2021 · Original source
Gotta disagree. Working on fixing the long term threat of super intelligent evil AI will not necessarily help you fix today's problems, but fixing today's problems will most certainly teach you something about fixing the long term threat, should it exist. I mean, I can't offer a complete criticism of the work done in AI risk, and I'm not going to sit here and play utilitarian table tennis with you over what extent we should worry about both the worst and least worst case at the same time. What I can do is choose where to put my money and time, and i think i make the reasonable choice not to spend too much time writing or building anything in the world of 'hypothetical AI', so to speak.
A late Byzantine shouldn’t have worried that cutesy fireworks were going to immediately lead to nukes. But instead of worrying that the fireworks would keep him up at night or explode in his face, he should have worried about giant cannons and the urgent need to remodel Constantinople’s defenses accordingly. If near-term AI risks are like worrying about fireworks keeping you awake at night and long-term AI risks like worrying about the Ottomans having cannons, then long-term AI worries are right. But if near-term AI risks are like worrying about the Ottomans having cannons and long-term AI risks are like worrying about nukes, then long-term AI worries are wrong.
Links For August
August 18, 2021 · Original source
20: How are resources in effective altruism allocated across issues? Most of the money goes to normal causes (eg global health) but most of the people go into meta-level concerns and weird causes (eg AI risk). Probably makes sense in the context of which problems are money-constrained vs. people-constrained and what the rest of the world is doing. Total amount of resources is probably on the order of $400 million/year and 2,000 highly engaged people.
Inline links: How are resources in effective altruism allocated across issues?
Absurdity Bias, Neom Edition
August 04, 2022 · Original source
But isn’t the absurdity heuristic a cognitive bias? Didn’t lots of true things sound absurd before they turned out to be true (eg evolution, quantum mechanics)? Don’t I specifically believe in things many people have found self-evidently absurd (eg the multiverse, AI risk)? Shouldn’t I be more careful about “this sounds silly to me, so I’m going to make fun of it”?
Inline links: isn’t the absurdity heuristic a cognitive bias
Book Review: What We Owe The Future
August 23, 2022 · Original source
Eli Lifland replies that sometimes long-termism and near-termism make different predictions; you need long-termism to robustly prioritize x-risk related charities. I am not sure I agree; his near-termist analysis still finds that AI risk is most cost-effective; the only thing even close is animal welfare, which many people reject based on not caring about animals. I think marginal thinking and moral parliament concerns can get you most of the way to an ideal balance of charitable giving without long-termism (see here for more), and that common-sense principles like “it would be extra bad if humanity went extinct” can get you the rest of the way.
Inline links: Eli Lifland replies
In Continued Defense Of Effective Altruism
November 28, 2023 · Original source
And I notice that the tiny handful of people capable of caring about 200,000 people dying of neglected tropical diseases are the same tiny handful of people capable of caring about the next pandemic, or superintelligence, or human extinction. I think the common skill is trying to analyze what causes are important logically. Surprisingly, this comes up with different results than the usual strategy of caring about whatever is trending on Twitter today! Some of those results are broadly popular, like saving 200,000 lives. Other results are broadly unpopular, like worrying about AI risk. But both sets of results come from the same process, and the people who get them are natural allies in a world where most people can’t bring themselves to care about anything that isn’t current front-page news.
Open Philanthropy Project originally got one seat on the OpenAI board by supporting them when they were still a nonprofit; that later went to Helen Toner. I’m not sure how Tasha McCauley got her seat. Currently the provisional board is Bret Taylor, Adam D’Angelo, and Larry Summers. Summers says he “believe[s] in effective altruism” but doesn’t seem AI-risk-pilled. Adam D’Angelo has never explicitly identified with EA or the AI risk movement but seems to have sided with the EAs in the recent fight so I’m not sure how to count him.
Open Thread 306
December 11, 2023 · Original source
1: Comments of the week: Timothy Johnson tries to explain why Taylor Swift is such a big deal; Waldo explains why regulated businesses might sue their regulators even when trying to stay on their good side; and Scott Aaronson sets the record straight on his beliefs about AI risk. And I previously said I couldn’t find the source of a poll claiming that the median American estimated a 26% chance AI would kill all humans, but an alert reader found it here. Remember that ordinary people aren’t good at asserting probabilities, and also that medians don’t always present the full picture; see the link for more details.
Inline links: why Taylor Swift is such a big deal, why regulated businesses might sue their regulators even when trying to stay on their good side, sets the record straight on his beliefs about AI risk, found it here
Contra Stone On EA
May 30, 2024 · Original source
Try spotting existential risk prevention on here. I don’t think Stone can claim that an EA version of this chart wouldn’t look phenomenally different. But then what’s left of his argument? III. Effective altruists devote absolutely enormous amounts of mental energy and research costs to program assessment, measurement of effectiveness. Those studies yield usually-conflicting results with variable effect sizes across time horizons and model specifications, and tons of different programs end up with overlapping effect estimates. That is to say, the areas where EAist style program evaluations are most compelling are areas where we don’t need them: it’s been obvious for a long time how to reduce malaria deaths, program evaluations on that front have been encouraging and marginally useful, but not gamechanging. On the other hand, in more contestable areas, EAist style program evaluations don’t really yield much clarity. It’s very rare that a program evaluation gets published finding vastly larger benefits than you’d guess from simple back-of-the-envelope guesswork, and the smaller estimates are usually because a specific intervention had first-order failure or long-run tapering, not because “actually tuberculosis isn’t that bad” or something like that. Those kinds of precise program-delivery studies are actually not an EAist specialty, but more IPA’s specialty. My second critique, then is this: there is no evidence that the toolkit and philosophical approach EAists so loudly proclaim as morally superior actually yields any clarity, or that their involvement in global efforts is net-positive vs. similar-scale donations given through near-peer organizations. The IPA mentioned here is Innovations For Poverty Action, a group that studies how to fight poverty. They’re great and do great work. But IPA doesn’t recommend top charities or direct donations. Go to their website, try to find their recommended charities. Unless I’m missing something, there are none. GiveWell does have recommended charities - including ones that they decided to recommend based on IPA’s work - and moves ~$250 million per year to them. If IPA existed, but not GiveWell, the average donor wouldn’t know where to donate, and ~$250 million per year would fail to go to charities that IPA likes. I think from the perspective of people who actually work within this ecosystem, Stone’s concern is like saying “Farms have already solved the making-food problem, so why do we need grocery stores?” (also, effective altruism funds IPA) I’m focusing on IPA here because Stone brought them up, but I think EA does more than this. I don’t think there’s an IPA for figuring out whether asteroid deflection is more cost-effective than biosecurity, whether cow welfare is more effective than chicken welfare, or figuring out which AI safety institute to donate to. I think this is because IPA is working on a really specific problem (which kinds of poverty-related interventions work) and EA is working on a different problem (what charities should vaguely utilitarian-minded people donate to?) These are closely related questions but they’re not the same question - which is why, for example, IPA does (great) research into consumer protection, something EA doesn’t consider comparatively high-impact. And I’m still focusing on donation to charity, again because it’s what Stone brought up, but EA does other things - like incubating charities, or building networks that affect policy. IV. Let’s skip farm animal welfare for a second and look at the next few: Global Aid, “Effective Altruism,” potential AI risks, biosecurity, and global catastrophic risk. These are all definitely disproportionate areas of EAist interest. If you google these topics, you will find a wildly disproportionate number of people who are EAist, or have sex at EAist orgies, or are the friends of people who have sex at EAist orgies. These really are some of the unique social features of EAism. And they largely amount to subsidizing white collar worker wages. I’m sorry but there’s no other way to slice it: these are all jobs largely aimed at giving money to researchers, PhD-holders, university-adjacent-persons, think tanks, etc. That may be fine stuff, but the whole pitch of effective altruism is that it’s supposed to bypass a lot of the conventional nonprofit bureaucracy and its parasitism and just give money to effective charities. But as EAism as matured into a truly unique social movement, it is creating its own bureaucracy of researchers, think tanks, bureaucrats… the very things it critiqued. Suppose an EA organization funded a cancer researcher to study some new drug, and that new drug was a perfect universal cure for cancer. Would Stone reject this donation as somehow impure, because it went to a cancer researcher (a white-collar PhD holder)? EA gives hundreds of millions of dollars directly to malaria treatments that go to the poorest people in the world. It’s also one the main funders of GiveDirectly, a charity that has given money ($750 million so far) directly to the poorest people in the world. But in addition to giving out bednets directly, it sometimes funds malaria vaccines. In addition to giving to poor Africans, it also funds the people who do the studies to see whether giving to poor Africans works. Some of those are white-collar workers. EA has never been about critiquing the existence of researchers and think tanks. In fact, this is part of the story of EA’s founding. In 2007, the only charity evaluators accessible by normal people rated charities entirely on how much overhead they had - whether the money went to white-collar people or to sympathetic poor recipients. EAs weren’t the first to point out that this was a very weak way of evaluating charities. But they were the first to make the argument at scale and bring it into the public consciousness, and GiveWell (and to some degree the greater EA movement) were founded on the principle of “what if there was a charity evaluator that did better than just calculate overhead?” In accordance with this history, if you look on Giving What We Can’s List Of Misconceptions About Effective Altruism, their #1 Misconception about about charity evaluation is that “looking at a charity’s overhead costs is key to evaluating its effectiveness”. This is another part of my argument that EA is more than just IPA++. For years, the state of the art for charity evaluators was “grade them by how much overhead they had”. IPA and all the great people working on evidence-based charity at the time didn’t solve that problem - people either used CharityNavigator or did their own research. GiveWell did solve that problem, and that success sparked a broader movement to come up with a philosophy of charity that could solve more problems. Many individuals have always had good philosophies of charity, but I think EA was a step change in doing it at scale and trying to build useful tools / a community around it. V. You could of course say AI risk is a super big issue. I’m open to that! But surely the solution to AI risk is to invest in some drone-delivered bombs and geospatial data on computing centers! The idea that the primary solution here is going to be blog posts, white papers, podcasts, and even lobbying is just insane. If you are serious about ruinous AI risk, you cannot possibly tell me that the strategy pursued here is optimal vs. say waiting until a time when workers have all gone home and blowing up a bunch of data centers and corporate offices. In particular terrorism as a strategy may be efficient since explosives are rather cheap. To be clear I do not support a strategy of terrorism!!!! But I am questioning why AI-riskers don’t. Logically, they should. I think if you have to write in bold with four exclamation points at the end that you’re not explicitly advocating terrorism, you should step back and think about your assumptions further. So: Should people who worry about global warming bomb coal plants? Should people who worry that Trump is going to destroy American democracy bomb the Republican National Convention? Should people who worry about fertility collapse and underpopulation bomb abortion clinics? EAs aren’t the only group who think there are deeply important causes. But for some reason people who can think about other problems in Near Mode go crazy when they start thinking about EA. (Eliezer Yudkowsky has sometimes been accused of wanting to bomb data centers, but he supports international regulations backed by military force - his model is things like Israel bombing Iraq’s nuclear program in the context of global norms limiting nuclear proliferation - not lone wolves. As far as I know, all EAs are united against this kind of thing.) There are three reasons not to bomb coal plants/data centers/etc. The first is that bombing things is morally wrong. I take this one pretty seriously. The second is that terrorism doesn’t work. Imagine that someone actually tried to bomb a data center. First of all, I don’t have statistics but I assume 99% of terrorists get caught at the “your collaborator is an undercover fed” stage. Another 99% get eliminated at the “blown up by poor bomb hygiene and/or a spam text message” stage. And okay, 1/10,000 will destroy a datacenter, and then what? Google tells me there are 10,978 data centers in the world. After one successful attack, the other 10,977 will get better security. Probably many of these are in China or some other country that’s not trivial for an American to import high explosives into. The third is that - did I say terrorism didn’t work? I mean it massively massively backfires. Hamas tried terrorism, they frankly did a much better job than we would, and now 52% of the buildings in their entire country have been turned to rubble. Osama bin Laden tried terrorism, also did an impressive job, and the US took over the whole country that had supported him, then took over an unrelated country that seemed like the kinds of guys who might support him, then spent ten years hunting him down and killing him and everyone he had ever associated with. One f@#king time, a handful of EAs tried promoting their agenda by committing some crimes which were much less bad than terrorism. Along with all the direct suffering they caused, they destroyed EA’s reputation and political influence, drove thousands of people away from the movement, and everything they did remains a giant pit of shame that we’re still in the process of trying to climb our way out of. Not to bang the same drum again and again, but this is why EA needs to be a coherent philosophy and not just IPA++. You need some kind of theory of what kinds of activism are acceptable and effective, or else people will come up with morally repugnant and incredibly idiotic plans that will definitely backfire and destroy everything you thought you were fighting for. EA hasn’t always been the best at avoiding this failure mode, but at least we manage to outdo our critics. VI. Stone moves on to animal welfare: It’s important to grasp that [caring about animals] is, in evolutionary terms, an error in our programming. The mechanisms involved are entirely about intra-human dynamics (or, some argue, may also be about recognizing the signs of vulnerable prey animals or enabling better hunting). Yes humans have had domestic animals for quite a long time, but our sympathetic responses are far older than that. We developed accidental sympathies for animals and then we made friends with dogs, not vice versa. Again, this is part of why I think it’s useful to have people who think about philosophy, and not just people who do RCTs. People having kids of their own instead of donating to sperm banks is in some sense an “error” in our evolutionary program. The program just wanted us to reproduce; instead we got a bunch of weird proxy goals like “actually loving kids for their own sake”. Art is another error - I assume we were evolutionarily programmed to care about beauty because, I don’t know, flowers indicate good hunting grounds or something, not because evolution wanted us to paint beautiful pictures. Anyone who cares about a future they will never experience, or about people on far off continents who they’ll never meet, is in some sense succumbing to “errors” in their evolutionary programming. Stone describes the original mechanisms as “about intra-human dynamics”, but this is cope - they’re about intra-tribal dynamics. Plenty of cultures have been completely happy to enslave, kill, and murder people outside their tribes, and nothing in their evolutionary mechanism has told them not to. Does Stone think this, too, is an error? At some point you’ve got to go beyond evolutionary programming and decide what kind of person you want to be. I want to be the kind of person who cares about my family, about beauty, about people on other continents, and - yes - about animal suffering. This is the reflective equilibrium I’ve landed in after considering all the drives and desires within me, filtering it through my ability to use Reason, and imagining having to justify myself to whatever God may or may not exist. Stone suggests EAs don’t have answers to a lot of the basic questions around this. I can recommend him various posts like Axiology, Morality, Law, the super-old Consequentialism FAQ, and The Gift We Give To Tomorrow, but I think they’ll only address about half of his questions. The other half of the answers have to come from intuition, common sense, and moral conservatism. This isn’t embarrassing. Logicians have discovered many fine and helpful logical principles, but can’t 100% answer the problem of skepticism - you can fill in some of the internal links in the chain, but the beginning and end stay shrouded in mystery. This doesn’t mean you can ignore the logical principles we do know. It just means that life is a combination of formally-reasonable and not-formally-reasonable bits. You should follow the formal reason where you have it, and not freak out and collapse into Cartesian doubt where you don’t. This is how I think of morality too. Again, I really think it’s important to have a philosophy and not just a big pile of RCTs. Our critics make this point better than I ever could. They start with “all this stuff is just common sense, who needs philosophy, the RCTs basically interpret themselves”, then, in the same essay, digress into: If I wanted to do this stuff, I would try terrorism.
Inline links: Giving What We Can’s List Of Misconceptions About Effective Altruism, Israel bombing Iraq’s nuclear program, I take this one pretty seriously, a spam text message, Axiology, Morality, Law, Consequentialism FAQ, The Gift We Give To Tomorrow
Highlights From The Comments On Fatima
October 24, 2025 · Original source
But if He does try to trick people, He should succeed. I can’t say either of these two things with confidence. Doesn’t the Biblical God sort of try to trick Abraham into thinking he’s going to have to sacrifice his son? And what is God, anyway? Isn’t the whole world a product of God? Does the existence of mirages in the desert count as “God trying to trick people”? Does that fact that we know there are mirages imply that God failed? Still, Ethan’s take on the “sun” miracle of Fatima seems like an unusually clear-cut case of God trying to trick people and failing, and I’m uncomfortable with it. You can always add more overfitting. God’s goal was for the crowds at Fatima to be fooled, but then for Dalleur (2021) to figure it out, and so He achieved His goal perfectly. Okay. But speaking of overfitting… If I understand Ethan right, Fatima was an objective omnidirectional light show, plus a unidirectional heat ray. Ghiaie was a spotlight-shaped unidirectional lightshow. Benin City was a subjective omnidirectional light show limited to a single field, plus an objective unidirectional heat ray. God implemented all of these miracles in completely different ways. Why? Inscrutable God reasons. This isn’t a terrible answer. People often do things for reasons I can’t explain - if I could predict Trump’s behavior, my stock market returns would be much higher. And surely God, as a being with motives and knowledge far beyond my ken, should be even more incomprehensible. But there was an interesting recent Notes debate about a Bentham Bulldog’s post. BB said that atheists had many problems - how was the world created? how do you overcome skepticism? what happened at Fatima? - whereas theism only has one problem - the problem of evil. Evil is a big problem, but it’s at least nice to only have one. Some of the commenters - and I can no longer find the comment I liked anymore, but don’t take this as an original insight from me - pointed out that this is cheap. If you are an atheist, you need to answer many how questions. How did the miracle at Fatima happen? If you try to explain it with natural laws - for example, gravity - it’s fair for an interlocutor to point out that gravity can’t do that; it can only make things fall. If you’re a theist, you have a free option to convert any how question to a why question. How? Because God did it! Your interlocutor can’t object, because we know God can do anything. But in exchange, you now have a why question - why did God do that, and not something else? The sum of all why question - the fact that the real world doesn’t look like it was optimized for some specific plausible motive like goodness - is the problem of evil. Thus, it is exactly equivalent to all the inconvenient “how” questions you hoped you’d avoided. The commenter sarcastically compared this to an attempt to sweep all scientific anomalies under the rug as “the problem of uncharacteristicness”. How did Fatima happen? “Well, it must have been produced by laws of physics, so there!” But the sun spinning and dancing through the sky is hardly what you would expect from the laws of physics. “Yeah, whatever, that’s just the ‘problem of uncharacteristicness’, we’ve already priced that one in, at least we only have one problem!” This made me more attuned to questions of God’s motives. Yeah, it is kind of weird that God would create the same miracle three different ways, and we don’t know why. Yeah, it is kind of weird that God would try to trick people into thinking a non-sun-object was the sun, then let a few smart people working years later see through the deception. Are these problems of motive exactly as problematic for the theist as 70,000 people seeing the sun do impossible things is for the atheist? My gut answer is no. Should I trust my gut? Dylan: In Defense Of Evan Harkness-Murphy Evan wrote the original response to Ethan, before I got involved in the debate. I was a bit harsh on him, saying that his part about the child-seers was fine, but calling his investigation of the sun miracle superficial and unfairly dismissive. Dylan of Chaotic Neutral writes In Defense Of Evan Harkness-Murphy, and Evan additionally defends himself here. Before getting to Dylan’s post - yeah, I was unfair to Evan (partly this is because my brain has trouble remembering that Ethan Muse and Evan Murphy are two different people). In particular, I described his hypothesis on the child-seers as being that they “confabulated” their visions, a term that Evan took great pains to disclaim in his actual post. I was thinking of a broader definition of “confabulation” that includes hallucination-like phenomena - but Evan was right that if I had read his post carefully, I wouldn’t have used the specific word he said he was against. I mostly just skimmed it to see if he had a really good explanation for the sun miracle thing, then got annoyed when he didn’t. But Dylan has additional complaints. He writes: Evan DID give this miracle the attention it deserved. He spent 18 hours researching and writing his article, presenting much of the same evidence and coming to many of the same conclusions that Scott did, and he did it as an ordinary citizen with a “day job” and in a household that “does not possess a dishwashing machine.” What more could you ask of a skeptical individual!? Unlike myself and the other lazy skeptics, he actually did respect this miracle claim enough to do a proper investigation. And towards the end, yes, he decided to wrap up early […] To criticize Evan’s conduct here in this miracle debate is to set an extremely high bar that cannot possibly be met by the overwhelming majority of the skeptical community. Such exacting standards will ultimately only serve to discourage diligent skepticism like Evan’s and incentivize lazy skepticism like mine. I have two partial defenses of my own actions. First, I think the majority of those 18 hours were spent on the child-seer section, which I acknowledged was good. I didn’t care about that part. To me, the trouble of explaining how three children can say in a convincing and honest-sounding and semiconsistent way that they saw the Virgin Mary is a rounding error compared to the trouble of explaining how 70,000 people can say in a convincing and honest-sounding and semiconsistent way that they saw the sun fall from the sky. But second, I think Dylan is arguing that Evan should get an A for effort. I agree. He put in a lot of work, he adhered to good scholarly principles, and he hit all of the beats that a skeptical explanation is supposed to hit. The only thing he didn’t do, from my perspective, is defuse the fact that the Fatima miracle is extremely creepy, and I have no idea what to do with it, and I can’t fit it into my ontology. Evan’s only attempt to defuse the miracle was that it was a hallucination or illusion or something. This is a reasonable conjecture, but for me it was already priced in - as soon as you hear about a miracle, the obvious next step is “well, maybe it was a hallucination or illusion or something”. I didn’t feel like his piece added anything extra. Generously, some of his tangential points - like that Garrett and Almeida weren’t the perfect skeptics they are sometimes portrayed as - might have defused 1% of my discomfort. I think a reasonable conclusion for this would have been “I’ve rehearsed the obvious arguments for why it is possible to be skeptical of anything, I’ve found some tangential facts that maybe remove 1% of the mystery, but man, I don’t know, this really needs lots more investigation”. My research hardly provided any kind of brilliant omni-solution, but I think that learning about the Ghiaie/Benin/Lubbock/Medjugorge followup miracles and the Redditor testimonies each defused about 15% of my reluctance to accept Fatima as natural, and the fire kasina + Khomeini stuff defused another 10%, to the point where I’m only about 60% as confused and unhappy as when I started. I hope I correctly signposted this level of success/failure to the reader. On Miracles Other responses tried to assert a general point that we should always disbelieve miracles. I. Eugene Earnshaw writes that We Do Not Need To Care About Miracles. If I understand his argument right: there are many examples of anomalous phenomena (eg crop circles) and stage magic (eg sawing a woman in half). When we don’t know how these are done, they seem impossible, and (almost) no amount of armchair reasoning can produce a plausible explanation. But in many cases, we have eventually figured them out - some “white hat” crop circlers explain how they make their seemingly-impossible patterns, and some magicians publish explanations of their tricks. After the fact, we can see how these seemingly-impossible things followed natural law after all. So we shouldn’t worry too much each time we encounter a new miracle that hasn’t yet been explained. Okay, but - suppose that the Pope said “I’m tired of convincing you people the normal ways, I’m going to start blowing up mountains”, and pointed his papal staff at Mt. Everest, and it exploded. And then we asked him to repeat the performance, and he did so as many times as we asked him, again and again. Would we shrug and say “Nothing to see here, I’m sure there’s some reasonable explanation”? If the miracle were sufficiently convincing, we would either believe it, or at least think it pointed at something interesting (maybe the Vatican obtained super-nukes and is hiding them under mountains and choreographing their detonations - but this would be pretty important and very different from “nothing to see here”). Ben Landau-Taylor gives a related answer, reminding us that meteorites used to be dismissed on exactly these grounds. The science of the day didn’t allow for non-planet objects to be in space, so rocks falling from the sky was every bit as weird as the sun dancing and changing colors. “When President Jefferson was told that Professors Silliman and Kingsley had described a fall of stones from the sky at Weston, in Connecticut, he remarked: ‘It is easier to believe that two Yankee professors will lie than to believe that stones will fall from heaven.’” In the end, I think we just get back to regular Bayesianism. We have two hypotheses: First, that the world acts entirely according to natural law. Second, that sometimes it includes divine intervention (or very surprising natural laws that we wouldn’t have predicted beforehand). We start with a high prior on the first hypothesis based on our long history of seeing only natural events. When we see evidence that is more likely on the second hypothesis than the first, we update in favor of it. We should remember that “more likely on the second hypothesis than the first” is full of pitfalls - on the first hypothesis, it’s likely that there will be many skilled fraudsters and stage magicians, so even very strange-seeming anomalies might not be very unlikely under it. Still, at the point where the Pope starts blowing up mountains, maybe you think it’s pretty unlikely that stage magic could accomplish this, and you update a little. II. Omne Bonum makes a different point: there are many possible miracles. Most do not occur. Yes, a few of them do. But can we be sure it’s above the background rate? Even if there are no true miracles, you’ll get one-in-a-million coincidences one-millionth of the time. If you’re not good at accounting for the 999,999 failures - and people aren’t - this will look impressive. Against this, what is the base rate for the sun changing color and dropping out of the sky, at the precise time that child-seers prophecied a miracle would occur? Seems lower than one in a million. Impossible things should never happen. Something as simple as my pen vanishing from my desk, in plain sight, while I am looking straight at it, should completely demolish all of my priors against miracles and make me near-certain that something beyond normal physical law is going on - or that I’m crazy, or dreaming, or something other than just “well it was a coincidence”. III. FLWAB takes on Hume’s argument against miracles (see also Kenny Easwaran here), which - sorry, I realize it’s suspicious to say this about a famous philosopher - is extremely bad. Hume argues that a miracle is a violation of natural law. And a natural law is something that is always true. But since it’s always true, it can’t be violated. And if we eventually confirmed that it was violated, then we were wrong about it being a natural law. Which means its violation wasn’t even a real miracle anyway. This seems to be a purely semantic argument. We know that the Red Sea usually stays in one place. But suppose Moses lifts his staff and parts the Red Sea, and that all of this is very convincing (we witness it personally, we measure the sea with various instruments, etc). I think Hume would have to say that we have disproven the natural law “the Red Sea usually stays in one place” - but only in favor of a new natural law “the Red Sea stays in one place except when Moses raises his staff”. And since we have never observed a violation of this new natural law, no miracle has occurred! Against this, we can call the way things work 99.999% of the time, when God isn’t acting directly, and when everything is proceeding via predictable material patterns “natural law”, and the very rare deviations that only occur in the presence of God or other extremely holy figures “miracles”. If for some reason you hate that terminology, come up with a new word, “shmiracle”, for the abnormal phenomena that only occur secondary to God’s direct intervention, and then we can argue whether shmiracles exist. IV. Why am I insisting on this so hard? This question of miracles is no different from every other question, where confirmation bias is a part of normal Bayesian reasoning. If you believe that vaccines don’t cause autism, then any given study showing that they do is likely to be a fraud or a mistake - especially given the history of such frauds, and the political pressures for producing them. But you gained your belief that vaccines don’t cause autism through some normal amount of evidence, and if the evidence that they did cause it ever become truly overwhelming, you would switch sides. The key skill of rationality is to know when to update your beliefs how much. These arguments feel like sleights-of-hand arguing that you can avoid ever updating on this question. I don’t think Bayesian reasoning provides an excuse for this. I think some of these arguments attempt to make an objection that the prior probability of miracles is zero, and so no matter how much evidence you get, you can never update towards them. But the prior probability of miracles isn’t zero unless either the prior probability of God’s existence is zero, or the probability that God intervenes in the universe is zero. I don’t know any infinitely-convincing argument for either of these points, so I think miracles have a prior probability above zero, which means we have to treat them the same as any other hypothesis. Yes, we will need many extra guardrails and cautions and good heuristics to prevent ourselves from getting bamboozled by the pitfalls that lurk in this area in particular. But that’s true of everything! You also need extra guardrails and cautions and heuristics to prevent yourself from getting bamboozled by scientific studies! There’s no substitute for doing the work. Actual Highlights From The Actual Comments Josh (blog) writes: I’d add that we have at least one verified case where a sun miracle was occuring, and an actual group of fedora wearing atheists were present with a modified telescope, and did not see anything interesting. >> “At the Conyers site, the Georgia Skeptics group set up a telescope outfitted with a vision-protecting Mylar solar filter, and on one occasion I participated in the experiment. Becky Long, president of the organization, stated that more than two hundred people had viewed the sun through one of the solar filters and not a single person saw anything unusual (Long 1992, 3; see figure 1).” https://centerforinquiry.s3.amazonaws.com/wp-content/uploads/sites/29/2009/11/22164423/p14.pdf Funny, but they don’t provide information like whether people were seeing sun miracles at the exact moment the telescope was being used, or whether anyone who could see a sun miracle without the telescope switched to using the telescope and then it stopped. They just say they brought a telescope to a Marian site where some people had seen sun miracles at some point. Even if they clarified that some people had used the telescope while seeing a sun miracle and had it immediately stop miracle-ing, I don’t think this would update me very much. We know it’s not the real sun (Ethan says fake sun, I say subjective phenomenon), and we know the non-Fatima miracles aren’t objective (Ethan says only Fatima was objective, I say none of them were objective). John Schilling writes: Twenty-nine *thousand* words on this subject, and none of them are “unidentified”, “flying”, or “object”. Well, OK, there are a few uses of that last, but in the strained phrasing of “UFO-like object”, as if we are preemptively discounting the possibility that sun miracles are actually UFOs. Sun miracles are actually UFOs, full stop. Not “flying saucers”, not “alien spaceships”, maybe “divine miracles”, but definitely “unidentified flying objects”. We invented that last phrase for a reason, and this is exactly that reason. Which means, the thing I learned from this is that the younglings have completely forgotten all that was learned in the Before Times about UFOs. And that, in this context, Scott is a youngling - UFOs seem to have faded from pop culture in the 1990s. Thanks for making me feel old, Scott :-) With the benefit of age and experience, I read the first few paragraphs, made the tentative conclusion that this was almost certainly [see section 6], but figured Scott wouldn’t be doing this deep a dive if it was that simple. And here we are. It probably is just that simple, and now we can back that up with a fairly exhaustive look at the alternatives. For which, unironically, thank you Scott. It’s good to sometimes double-, triple-, and quadruple-check the obvious conclusion. But for those of us who grew up in the 1980s, who were “rationalists” when rationalism hadn’t been invented and we had to call ourselves “skeptics”, UFOs were as important a subject of rationalist/skeptical inquiry as is AI risk today (and for about the same reason). People learned an awful lot in those days. One of those things is that most people don’t spend much time really looking at the sky and will consistently fail to recognize even slightly-unusual phenomena, like the sun partially veiled by clouds. And the other, more important thing is that when presented with an image they don’t recognize, people will very predictably see what their culture has taught them to expect to see. In 1880s-1890s America, any weird thing in the sky was clearly a fantastic airship, built by some mad scientist out of a Jules Verne novel, and was perceived with a wealth of surrounding detail all aligned with that model. 1950s-1980s America, the same things were clearly “flying saucers”, fantastic alien spaceships piloted by little green or grey men, with the same level of impossible detail. And anywhere you’ve got ten thousand devout Catholics fervently hoping to see a Miracle involving the Sun, and the weather makes the sun look a bit wonky... For an old-school skeptical experiment at understanding this effect, https://centerforinquiry.s3.amazonaws.com/wp-content/uploads/sites/29/1980/04/22165441/p34.pdf TL, DR, a gathering of UFO enthusiasts expecting to see a flying saucer in the night sky, are presented with thirty seconds of a monochromatic point source of light at ground level, stationary and unchanging except for one brief interruption. What is perceived, is an object high in the sky with finite angular size and geometric shape, of multiple colors, and conspicuously moving, all consistent with the pop-culture concept of a flying saucer and not some prankster with a spotlight. I considered discussing the UFO angle (the section heading would have been “Virgin Galactic”), but in the end I couldn’t justify it. Yes, the phenomenon is trivially a UFO (in the sense of a thing in the sky we don’t understand). But does this help us? When I think of UFOs, I think of people arguing about whether something was the planet Venus, or a weather balloon, or aliens. But Fatima obviously wasn’t Venus or a balloon (though, uh, see here for a dissenting take). And if it was aliens, you’d have to explain why they pretended to be the Virgin Mary and discussed a bunch of Catholic inside-baseball with a trio of child-seers for several months. So what’s left? When I asked John, he answered: UFOs, are just people seeing something they don’t understand and trying to interpret it by an overweighted, culturally-transmitted prior. Which differs from culture to culture. And that’s something we know a lot about. Which you seem to have independently rediscovered, but I can’t help thinking you’d have got there a lot faster if you’d had a proper map of the territory. A map which includes no aliens outside of the imaginary sort. Maybe one way to rescue the UFO connection is to say that there’s so much weirdness that we should be less willing to take any given example of weirdness on its own terms. I asked in the comments for other examples of miracles as compelling as Fatima. People suggested some of the better-verified reincarnation accounts, some of the better-verified UFO sightings, and some of the more spectacular psi phenomena. I don’t know if these are all exactly as strong as Fatima, but I think many of them are closer to Fatima than to the traditional skeptical conception of an alcoholic liar asserting with zero evidence that he dun saw dem aliens one night. When viewing all of these anomalies as a gestalt, we can go four different directions: Individualized natural explanations. The UFOs were swamp gas and weather balloons. The reincarnation stories are toddlers who are naturally gifted at cold reading. Fatima was entoptic phenomena. Sea serpents are really big oarfish.
Inline links: an interesting recent Notes debate, In Defense Of Evan Harkness-Murphy, defends himself here, We Do Not Need To Care About Miracles, explain, gives, Omne Bonum, takes on Hume’s argument against miracles, here, confirmation bias is a part of normal Bayesian reasoning, blog, https://centerforinquiry.s3.amazonaws.com/wp-content/uploads/sites/29/2009/11/22164423/p14.pdf, writes, https://centerforinquiry.s3.amazonaws.com/wp-content/uploads/sites/29/1980/04/22165441/p34.pdf, see here, https://substackcdn.com/image/fetch/$s_!H6Qh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6087bbd3-2496-46b1-87e0-f96ce73c19d2_522x223.png