Astral Codex Ten

RCTs

17 min read

RCTs

Article

RCTs is a recurring concept in the Astral Codex Ten archive, appearing 4 times across 4 issues between April 30, 2021 and May 30, 2024. The archive places it in contexts such as “Wizardly tools (science, technology, RCTs)”; “There are dozens of positive double-blind RCTs of homeopathy”; “put the results of RCTs in context”. It most often appears alongside EA, Bay Area, COVID.

Metadata

  • Category: Concepts
  • Mention count: 4
  • Issue count: 4
  • First seen: April 30, 2021
  • Last seen: May 30, 2024

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Your Book Review: The Wizard And The Prophet
April 30, 2021 · Original source
A friend of mine who works in politics thinks there’s a third kind of archetype we seem to be missing in the Wizard/Prophet dichotomy – something like the "Engineer" who can tinker with complex, semi-broken systems using a mix of Wizardly tools (science, technology, RCTs) and Prophetic ones (grass-roots activism, behavioral and cultural change) to get them retuned and producing better long term outputs. Another in academia thinks genetically engineering everyone to be smarter is the only way to make real progress on the thornier, hairier systemic problems. Half the people I know in the Bay Area are convinced that democratic socialism is the true path forward; the other half are pretty sure that AI will eventually, not-too-distantly-from-now destroy everything, so other kinds of long term systems tinkering probably aren’t even worth worrying about. (It’s interesting to me that the realm of AI research is populated by highly educated, technocratic Wizard types, but while its tenor may have started out very Wizardly, it is now extremely Prophetic.)
Inline links: while its tenor may have started out very Wizardly, it is now extremely Prophetic
Highlights From The Comments On Ivermectin
November 23, 2021 · Original source
I think the main thing I want to cram into his head is how many pseudosciences that have to be false have really strong empirical literatures behind them. There are dozens of positive double-blind RCTs of homeopathy. I feel like I can explain what went wrong with these about a third of the time. The rest of the time, I’m as boggled as everyone else, and I just accept that the biggest studies by the most careful people usually don’t find effects, plus we should have a low prior on an effect since it’s crazy. This makes me pretty willing to shrug and say “Yeah, I have no idea what went wrong here, but a few big RCTs didn’t find an effect, plus I have a super-high prior for any new medical thing being false, so whatever, let’s move on”, which I admit is unvirtuous but I’m not sure how to avoid it.
Highlights From The Comments On Criticism Of Criticism Of Criticism
July 28, 2022 · Original source
RCTs are a hugely powerful tool for studying charitable interventions, for all the reasons you already know. But when you first get started, it’s really easy to mistake “the results of an RCT” for “the entire relevant truth”, which is the sort of mistake that can massively ruin lives (or waste hundreds of millions of dollars) if you have the power to make decisions but not the experience to know how to interpret the relevant evidence. I wrote the review not to talk people out of EA (I like EA and am involved in an RCT I think will really help add to our knowledge of how to do good!) but because I think being aware of this kind of shortcoming and when to look out for it is necessary to put the results of RCTs in context and use them in a way that’s more responsible than either “just go off vibes” or “use only numerical quantitative information and nothing else”.
Contra Stone On EA
May 30, 2024 · Original source
Try spotting existential risk prevention on here. I don’t think Stone can claim that an EA version of this chart wouldn’t look phenomenally different. But then what’s left of his argument? III. Effective altruists devote absolutely enormous amounts of mental energy and research costs to program assessment, measurement of effectiveness. Those studies yield usually-conflicting results with variable effect sizes across time horizons and model specifications, and tons of different programs end up with overlapping effect estimates. That is to say, the areas where EAist style program evaluations are most compelling are areas where we don’t need them: it’s been obvious for a long time how to reduce malaria deaths, program evaluations on that front have been encouraging and marginally useful, but not gamechanging. On the other hand, in more contestable areas, EAist style program evaluations don’t really yield much clarity. It’s very rare that a program evaluation gets published finding vastly larger benefits than you’d guess from simple back-of-the-envelope guesswork, and the smaller estimates are usually because a specific intervention had first-order failure or long-run tapering, not because “actually tuberculosis isn’t that bad” or something like that. Those kinds of precise program-delivery studies are actually not an EAist specialty, but more IPA’s specialty. My second critique, then is this: there is no evidence that the toolkit and philosophical approach EAists so loudly proclaim as morally superior actually yields any clarity, or that their involvement in global efforts is net-positive vs. similar-scale donations given through near-peer organizations. The IPA mentioned here is Innovations For Poverty Action, a group that studies how to fight poverty. They’re great and do great work. But IPA doesn’t recommend top charities or direct donations. Go to their website, try to find their recommended charities. Unless I’m missing something, there are none. GiveWell does have recommended charities - including ones that they decided to recommend based on IPA’s work - and moves ~$250 million per year to them. If IPA existed, but not GiveWell, the average donor wouldn’t know where to donate, and ~$250 million per year would fail to go to charities that IPA likes. I think from the perspective of people who actually work within this ecosystem, Stone’s concern is like saying “Farms have already solved the making-food problem, so why do we need grocery stores?” (also, effective altruism funds IPA) I’m focusing on IPA here because Stone brought them up, but I think EA does more than this. I don’t think there’s an IPA for figuring out whether asteroid deflection is more cost-effective than biosecurity, whether cow welfare is more effective than chicken welfare, or figuring out which AI safety institute to donate to. I think this is because IPA is working on a really specific problem (which kinds of poverty-related interventions work) and EA is working on a different problem (what charities should vaguely utilitarian-minded people donate to?) These are closely related questions but they’re not the same question - which is why, for example, IPA does (great) research into consumer protection, something EA doesn’t consider comparatively high-impact. And I’m still focusing on donation to charity, again because it’s what Stone brought up, but EA does other things - like incubating charities, or building networks that affect policy. IV. Let’s skip farm animal welfare for a second and look at the next few: Global Aid, “Effective Altruism,” potential AI risks, biosecurity, and global catastrophic risk. These are all definitely disproportionate areas of EAist interest. If you google these topics, you will find a wildly disproportionate number of people who are EAist, or have sex at EAist orgies, or are the friends of people who have sex at EAist orgies. These really are some of the unique social features of EAism. And they largely amount to subsidizing white collar worker wages. I’m sorry but there’s no other way to slice it: these are all jobs largely aimed at giving money to researchers, PhD-holders, university-adjacent-persons, think tanks, etc. That may be fine stuff, but the whole pitch of effective altruism is that it’s supposed to bypass a lot of the conventional nonprofit bureaucracy and its parasitism and just give money to effective charities. But as EAism as matured into a truly unique social movement, it is creating its own bureaucracy of researchers, think tanks, bureaucrats… the very things it critiqued. Suppose an EA organization funded a cancer researcher to study some new drug, and that new drug was a perfect universal cure for cancer. Would Stone reject this donation as somehow impure, because it went to a cancer researcher (a white-collar PhD holder)? EA gives hundreds of millions of dollars directly to malaria treatments that go to the poorest people in the world. It’s also one the main funders of GiveDirectly, a charity that has given money ($750 million so far) directly to the poorest people in the world. But in addition to giving out bednets directly, it sometimes funds malaria vaccines. In addition to giving to poor Africans, it also funds the people who do the studies to see whether giving to poor Africans works. Some of those are white-collar workers. EA has never been about critiquing the existence of researchers and think tanks. In fact, this is part of the story of EA’s founding. In 2007, the only charity evaluators accessible by normal people rated charities entirely on how much overhead they had - whether the money went to white-collar people or to sympathetic poor recipients. EAs weren’t the first to point out that this was a very weak way of evaluating charities. But they were the first to make the argument at scale and bring it into the public consciousness, and GiveWell (and to some degree the greater EA movement) were founded on the principle of “what if there was a charity evaluator that did better than just calculate overhead?” In accordance with this history, if you look on Giving What We Can’s List Of Misconceptions About Effective Altruism, their #1 Misconception about about charity evaluation is that “looking at a charity’s overhead costs is key to evaluating its effectiveness”. This is another part of my argument that EA is more than just IPA++. For years, the state of the art for charity evaluators was “grade them by how much overhead they had”. IPA and all the great people working on evidence-based charity at the time didn’t solve that problem - people either used CharityNavigator or did their own research. GiveWell did solve that problem, and that success sparked a broader movement to come up with a philosophy of charity that could solve more problems. Many individuals have always had good philosophies of charity, but I think EA was a step change in doing it at scale and trying to build useful tools / a community around it. V. You could of course say AI risk is a super big issue. I’m open to that! But surely the solution to AI risk is to invest in some drone-delivered bombs and geospatial data on computing centers! The idea that the primary solution here is going to be blog posts, white papers, podcasts, and even lobbying is just insane. If you are serious about ruinous AI risk, you cannot possibly tell me that the strategy pursued here is optimal vs. say waiting until a time when workers have all gone home and blowing up a bunch of data centers and corporate offices. In particular terrorism as a strategy may be efficient since explosives are rather cheap. To be clear I do not support a strategy of terrorism!!!! But I am questioning why AI-riskers don’t. Logically, they should. I think if you have to write in bold with four exclamation points at the end that you’re not explicitly advocating terrorism, you should step back and think about your assumptions further. So: Should people who worry about global warming bomb coal plants? Should people who worry that Trump is going to destroy American democracy bomb the Republican National Convention? Should people who worry about fertility collapse and underpopulation bomb abortion clinics? EAs aren’t the only group who think there are deeply important causes. But for some reason people who can think about other problems in Near Mode go crazy when they start thinking about EA. (Eliezer Yudkowsky has sometimes been accused of wanting to bomb data centers, but he supports international regulations backed by military force - his model is things like Israel bombing Iraq’s nuclear program in the context of global norms limiting nuclear proliferation - not lone wolves. As far as I know, all EAs are united against this kind of thing.) There are three reasons not to bomb coal plants/data centers/etc. The first is that bombing things is morally wrong. I take this one pretty seriously. The second is that terrorism doesn’t work. Imagine that someone actually tried to bomb a data center. First of all, I don’t have statistics but I assume 99% of terrorists get caught at the “your collaborator is an undercover fed” stage. Another 99% get eliminated at the “blown up by poor bomb hygiene and/or a spam text message” stage. And okay, 1/10,000 will destroy a datacenter, and then what? Google tells me there are 10,978 data centers in the world. After one successful attack, the other 10,977 will get better security. Probably many of these are in China or some other country that’s not trivial for an American to import high explosives into. The third is that - did I say terrorism didn’t work? I mean it massively massively backfires. Hamas tried terrorism, they frankly did a much better job than we would, and now 52% of the buildings in their entire country have been turned to rubble. Osama bin Laden tried terrorism, also did an impressive job, and the US took over the whole country that had supported him, then took over an unrelated country that seemed like the kinds of guys who might support him, then spent ten years hunting him down and killing him and everyone he had ever associated with. One f@#king time, a handful of EAs tried promoting their agenda by committing some crimes which were much less bad than terrorism. Along with all the direct suffering they caused, they destroyed EA’s reputation and political influence, drove thousands of people away from the movement, and everything they did remains a giant pit of shame that we’re still in the process of trying to climb our way out of. Not to bang the same drum again and again, but this is why EA needs to be a coherent philosophy and not just IPA++. You need some kind of theory of what kinds of activism are acceptable and effective, or else people will come up with morally repugnant and incredibly idiotic plans that will definitely backfire and destroy everything you thought you were fighting for. EA hasn’t always been the best at avoiding this failure mode, but at least we manage to outdo our critics. VI. Stone moves on to animal welfare: It’s important to grasp that [caring about animals] is, in evolutionary terms, an error in our programming. The mechanisms involved are entirely about intra-human dynamics (or, some argue, may also be about recognizing the signs of vulnerable prey animals or enabling better hunting). Yes humans have had domestic animals for quite a long time, but our sympathetic responses are far older than that. We developed accidental sympathies for animals and then we made friends with dogs, not vice versa. Again, this is part of why I think it’s useful to have people who think about philosophy, and not just people who do RCTs. People having kids of their own instead of donating to sperm banks is in some sense an “error” in our evolutionary program. The program just wanted us to reproduce; instead we got a bunch of weird proxy goals like “actually loving kids for their own sake”. Art is another error - I assume we were evolutionarily programmed to care about beauty because, I don’t know, flowers indicate good hunting grounds or something, not because evolution wanted us to paint beautiful pictures. Anyone who cares about a future they will never experience, or about people on far off continents who they’ll never meet, is in some sense succumbing to “errors” in their evolutionary programming. Stone describes the original mechanisms as “about intra-human dynamics”, but this is cope - they’re about intra-tribal dynamics. Plenty of cultures have been completely happy to enslave, kill, and murder people outside their tribes, and nothing in their evolutionary mechanism has told them not to. Does Stone think this, too, is an error? At some point you’ve got to go beyond evolutionary programming and decide what kind of person you want to be. I want to be the kind of person who cares about my family, about beauty, about people on other continents, and - yes - about animal suffering. This is the reflective equilibrium I’ve landed in after considering all the drives and desires within me, filtering it through my ability to use Reason, and imagining having to justify myself to whatever God may or may not exist. Stone suggests EAs don’t have answers to a lot of the basic questions around this. I can recommend him various posts like Axiology, Morality, Law, the super-old Consequentialism FAQ, and The Gift We Give To Tomorrow, but I think they’ll only address about half of his questions. The other half of the answers have to come from intuition, common sense, and moral conservatism. This isn’t embarrassing. Logicians have discovered many fine and helpful logical principles, but can’t 100% answer the problem of skepticism - you can fill in some of the internal links in the chain, but the beginning and end stay shrouded in mystery. This doesn’t mean you can ignore the logical principles we do know. It just means that life is a combination of formally-reasonable and not-formally-reasonable bits. You should follow the formal reason where you have it, and not freak out and collapse into Cartesian doubt where you don’t. This is how I think of morality too. Again, I really think it’s important to have a philosophy and not just a big pile of RCTs. Our critics make this point better than I ever could. They start with “all this stuff is just common sense, who needs philosophy, the RCTs basically interpret themselves”, then, in the same essay, digress into: If I wanted to do this stuff, I would try terrorism.
Inline links: Giving What We Can’s List Of Misconceptions About Effective Altruism, Israel bombing Iraq’s nuclear program, I take this one pretty seriously, a spam text message, Axiology, Morality, Law, Consequentialism FAQ, The Gift We Give To Tomorrow
Morality is tough. Converting RCTs - let alone the wide world of things we don’t have RCTs on yet - into actionable suggestions is tough. Many people have tried this. Some have succeeded very well on their own. Effective altruism is a community of people working on this problem together. I’m grateful to have it.