Astral Codex Ten

Bounded Distrust

10 min read

Bounded Distrust

Article

Bounded Distrust is a recurring publication in the Astral Codex Ten archive, appearing 4 times across 4 issues between March 01, 2022 and January 08, 2025. The archive places it in contexts such as “the other relevant ACX post here is Bounded Distrust”; “Related: Bounded Distrust , Moderation Is Different From Censorship”; “Bounded Distrust also helps explain what I mean here”. It most often appears alongside New York Times, ACX, Matt Yglesias.

Metadata

  • Category: Publications
  • Mention count: 4
  • Issue count: 4
  • First seen: March 01, 2022
  • Last seen: January 08, 2025

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Ukraine Warcasting
March 01, 2022 · Original source
(Can we do better than this level of agnosticism? Someone suggested that the intelligence community might suck at the sort of small-state terrorism work it’s been asked to do the past few decades, but that “infiltrating Russia” is kind of its bread and butter and a big part of its institutional DNA. Maybe we should trust it more on Great Power conflict than on tinpot dictator stuff? Maybe the other relevant ACX post here is Bounded Distrust?)
Inline links: Bounded Distrust
The Media Very Rarely Lies
December 22, 2022 · Original source
Related: Bounded Distrust, Moderation Is Different From Censorship
Inline links: Bounded Distrust, Moderation Is Different From Censorship
Trying Again On Fideism
February 15, 2023 · Original source
Eliezer Yudkowsky’s position is Let Them Debate College Students. I’m not a college student, but I’m not Anthony Fauci either, and I am known for blogging about extremely dignified ideas like the possibility that the terrible Harry Potter fanfiction My Immortal is secretly an alchemical allegory. I haven’t seen ivermectin advocates using “Scott takes this seriously enough to argue against it!” as an argument, and I have seen them getting angry about it and writing long responses trying to prove me wrong. Sometimes they have used me getting some points wrong as a positive argument, and I would be open to the argument that I failed in not arguing against it well enough that they couldn’t do that, but nobody has been making that argument, and if they did, then it would imply that people who are smarter than me should take over the job, which I endorse. III. I worry Scott Aaronson thinks I’m saying you shouldn’t trust the experts, and instead you should always think for yourself. I’m definitely not trying to say that. I’ve tried to be pretty clear that I think experts are right remarkably often, by some standards basically 100% of the time - I realize how crazy that sounds, and “by some standards” is doing a lot of the work there, but see Learning To Love Scientific Consensus for more. Bounded Distrust also helps explain what I mean here. I also try to be pretty clear that reasoning is extremely hard, it’s very easy to get everything wrong, and if you try to do it then a default option is to get everything wrong and humiliate yourself. I describe that happening to me here, and presumably it also happens to other people sometimes. What I do think is that “trust the experts” is an extremely exploitable heuristic, which leads everyone to put up a veneer of “being the experts” and demand that you trust them. I come back to this example again and again, but only because it’s so blatant: the New York Times ran an article saying that only 36% of economists supported school vouchers, with a strong implication that the profession was majority against. If you checked their sources, you would find that actually, it was 36% in favor, 19% against, 46% unsure or not responding. If you are too quick to seek epistemic closure because “you have to trust the experts”, you will be easy prey to people misrepresenting what they are saying. I come back to this example less often, because it could get me in trouble, but when people do formal anonymous surveys of IQ scientists, they find that most of them believe different races have different IQs and that a substantial portion of the difference is genetic. I don’t think most New York Times readers would identify this as the scientific consensus. So either the surveys - which are pretty official and published in peer-reviewed journals - have managed to compellingly misrepresent expert consensus, or the impressions people get from the media have, or “expert consensus” is extremely variable and complicated and can’t be reflected by a single number or position. And I genuinely think this is part of why ivermectin conspiracies took off in the first place. We say “trust science” and “trust experts”. But there were lots of studies that showed ivermectin worked - aren’t those science? And Pierre Kory MD, an specialist in severe respiratory illnesses who wrote a well-regarded textbook, supports it - isn’t he an expert? Isn’t it plausible that the science and the experts are right, and the media and the government and Big Pharma are wrong? This is part of what happens when people reify the mantras instead of using them as pointers to more complicated concepts like “reasoning is hard” and “here are the 28,491 rules you need to keep in mind when reading a scientific study.” IV. All of this still feels rambly and like it’s failing to connect. Instead, let me try describing exactly what I would advice I would give young people opening an Internet connection for the first time: You are not immune to conspiracy theories. You have probably developed a false sense of security by encountering many dumb conspiracy theories and feeling no temptation to believe them. These theories were designed to trap people very different from you; others will be aimed in your direction. The more certain you are of your own infallibility, the less aware you will be, and the worse your chances. The ones that get you won’t look like conspiracy theories to you (though they might to other people). When you run into conspiracy theories you don’t believe, feel free to ignore them. If you decide to engage, don’t mock them or feel superior. Think “there, but for the grace of God, go I.” Get a sense of what the arguments for the conspiracy theory look like - not from skeptics trying to mock them, but from the horse’s mouth - so you have a sense of what false arguments look like. Ask yourself what habits of mind it would have taken the people affected by the theory to successfully resist it. Ask yourself if you have those habits of mind. Yes? ARE YOU SURE? To a first approximation, trust experts over your own judgment. If people are trying to confuse you about who the experts are, then to a second approximation trust prestigious people and big institutions, including professors at top colleges, journalists at major newspapers, professional groups with names like the American ______ Association, and the government. You might ask: Don’t governments and other big institutions have biases? Won’t they sometimes be wrong or deceptive? And even if you’ve lucked into the one country and historical era where the government 100% tells the truth and the intellectuals have no biases, doesn’t someone need to keep the flame of suspicion alive so that it’s available to people in other, less fortunate countries and eras? The answer is: absolutely, yes, but also this is how conspiracy theories get you. They will claim that they are the special case where you need to take up the mantle of Galileo and Frederick Douglass and Jane Jacobs and all those people who stood up to the intellectual authorities and power structures of their own time. The whole point of “you are not immune to conspiracy theories” is that the evidence for them can sound convincing because something like it is sort of true. This is equally so for second-level claims like “prestigious institutions are fallible and biased”. Probably something like “make a principled precommitment never to disagree with prestigious institutions until you are at least 30 and have a graduate degree in at least one subject” would be good advice, but nobody would take that advice, and taking it too seriously might crush some kind of important human spirit, so I won’t assert this. But always have in the back of your mind that you live in a world where it’s sort of good advice. If you feel tempted to believe something that has red flags for being a conspiracy theory, at least keep track of the Inside vs. Outside View. Say “on the Inside View, this feels like the evidence is overwhelming; on the Outside View, it sounds like a classic conspiracy theory”. You don’t necessarily have to resolve this discomfort right away. You can walk around with an annoying knot in your beliefs, even if it’s not fun. Look for the strongest evidence against the idea. Keep in mind important possibilities like: Is it possible that everyone who disagrees with the idea is a bad mean cruel stupid person, but also, the idea really is false?
Inline links: Let Them Debate College Students, like the possibility that the terrible Harry Potter fanfiction My Immortal is secretly an alchemical allegory, Learning To Love Scientific Consensus, Bounded Distrust, happening to me here, ran an article, ARE YOU SURE?
All advice along the lines of “don’t do X unless you’re smart and sophisticated” is useless, because everyone believes themselves smart and sophisticated. Still, at some point, after a lot of experience and a few crises of faith, you might develop a skill something like Bounded Distrust, at which point it’s not necessarily instant epistemic suicide to suspend the second approximation.
Inline links: Bounded Distrust
On Priesthoods
January 08, 2025 · Original source
Priesthoods make things up differently from normal people. Even when they’re corrupt, they still have a reputation to maintain. I’ve written about this before at Bounded Distrust and The Media Very Rarely Lies. Because priests are so focused on their reputation, even their mistruths follow certain ritual purity laws. The typical non-priest who lies to support a political cause will repeat some un-fact-checked lurid anecdote, or some utterly idiotic misinterpretation of garbled data. But the deceits of priests are subtle and elegant. They’ll publish a study that observes the forms almost perfectly, then bury something in the footnotes which reveals that it’s irrelevant to any of the real world situations that people would expect it to be relevant to.
Inline links: Bounded Distrust, The Media Very Rarely Lies