Astral Codex Ten

ivmmeta

14 min read

ivmmeta

Article

ivmmeta is a recurring organization in the Astral Codex Ten archive, appearing 5 times across 5 issues between November 23, 2021 and February 20, 2023. The archive places it in contexts such as “ivmmeta does better and clearer science communication than everyone else”; “sidebar of the ivmmeta site”; “keeping all of the studies mentioned on ivmmeta, removing the ones I think are bad”. It most often appears alongside Alexandros Marinos, COVID, ivermectin.

Metadata

  • Category: Organizations
  • Mention count: 5
  • Issue count: 5
  • First seen: November 23, 2021
  • Last seen: February 20, 2023

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Highlights From The Comments On Ivermectin
November 23, 2021 · Original source
Let’s start with the negative comments. Leading pro-ivermectin website ivmmeta.com understandably disagreed with my fisking of them. They have a section where they respond to critics (see responses to Gideon Meyerowitz-Katz, to the BBC, to the parasitic worm hypothesis, and to someone named AT who they won’t explain further). I was honored to also get a response here. They write:
Inline links: Gideon Meyerowitz-Katz, the BBC, the parasitic worm hypothesis, someone named AT who they won’t explain further
The steps required to accept the strongyloides-mechanism-only conclusion are also extreme - we need to disregard the majority of outcomes occuring before steroid use, and disregard the strong treatment-delay response relationship which is contradictory. Figure 24 shows analysis by strongyloides prevalence.
Inline links: Figure 24
The third-party analysis that author references for the strongyloides theory is confounded by treatment delay — the high prevalence group has more early treatment trials, and the low prevalence group has more late treatment trials, i.e., the analysis reflects the greater efficacy of early treatment. More details can be found in the strongyloides section.
Inline links: strongyloides section
Pascalian Medicine
November 24, 2021 · Original source
We can go further. The same people behind ivmmeta.com have posted this “meta-analysis” of curcumin, a common spice and oft-mooted panacea:
But what’s true of curcumin is equally true of lots of other different compounds: zinc, hydroxychloroquine, quercetin, nigella sativa, melatonin…just going off the ones on the sidebar of ivmmeta.com, there are about thirty different things that have this same level of very early, very dubious super-promising COVID results. Some are expensive and some are dangerous, but I think about twenty of them are cheap and safe.
I think ivermectin doesn’t work. I think that it looks like it works, because it has lots of positive studies and a few big-name endorsements. But our current scientific method is so weak and error-prone that any chemical which gets raised to researchers’ attentions and studied in depth will get approximately this amount of positive results and buzz. Look through the thirty different chemicals featured on the sidebar of the ivmmeta site if you don’t believe me.
Open Thread 200
November 28, 2021 · Original source
5: Alexandros Marinos, whose pro-ivermectin views I argued against in the same comments post, has finally started a Substack and written up those views at length. Among his interesting findings are that keeping all of the studies mentioned on ivmmeta, removing the ones I think are bad, removing the ones ivmmeta itself thinks are bad, and removing the ones that leading anti-ivermectin researcher Gideon Meyerowitz-Katz thinks are bad - all give about the same relative risk result (by ivmmeta’s methodology), somewhere around 0.3 or 0.4 (Marinos thinks that my and ivmmeta’s exclusions are similar around 0.3, and GMK’s exclusions are different around 0.4, but this seems like splitting hairs to me, since all three are overwhelmingly positive by these standards). I think this is an interesting finding about how (at least when critiquing ivmmeta) it’s probably not worth arguing over which studies to include or not, so much as about the overall methodology for how we interpret the studies remaining.
Inline links: finally started a Substack and written up those views at length
Response To Alexandros Contra Me On Ivermectin
February 01, 2023 · Original source
The control groups in high-worm-prevalence-area studies had no more deaths than in low-worm-prevalence-area studies. If the worms were killing people in the control groups (who ivermectin was then saving in the treatment groups) you would expect more deaths. You can find arguments for all these points at the link. (One additional thing Alexandros does that I really like: he compares the Strongyloides hypothesis - as an attempt to explain why these studies keep getting such different results - to other hypotheses. For example, studies in Latin America get negative results more often than others. This really feels like confronting the real question. He finds that Latin American studies do find lower efficacy for ivermectin than the other mostly Asian studies, and hypothesizes that this is because ivermectin is very popular in Latin America, the “control” group illicitly takes it without telling the researchers, and so these studies are inadvertantly comparing two ivermectin groups. This is another clever and elegant theory. Unfortunately, the recent spate of negative American studies sink it6. Still, I agree there is a strong geographic element here; worms are one possible explanation, but there are others - including the scientific culture in different countries. I appreciate Alexandros highlighting how much this is true.) I asked Dr. Bitterman for his thoughts. He reiterates that although steroids are one major cause of Strongyloides hyperinfection, another is eosinopenia, a decrease in the immune cells that fight parasites. COVID can cause eosinopenia directly, so just because a COVID patient didn’t get steroids, or was only on steroids for a short period, doesn’t prove that the patient couldn’t have had hyperinfection. On the mixing of different sources to get Strongyloides prevalence data, he said: As mentioned in the paper, when available we attempted to granulate by regional prevalence. This was often not possible because robust data did not available for a given country. Brazil is a large country (and multiple different studies in our analysis were in Brazil) with variability and Paula was a robust study. We decided to attempt to granulate instead of stacking the Brazil countries with the same prevalence even though they are in very different regions. His re-analysis is a crude one, and he often switches between using that analysis and the ecological model study. At the time of out paper's publication, the ecological model was not available. I offered to re-do the whole analysis with that study's raw data with a mutually agreed upon methodology, which we have not fully ironed down yet. On the long delay before hyperinfection kills: I don't think it happens in 1 or 2 weeks. But 3-4 weeks (within almost all study durations) is certainly not unheard of (again, without being treated). Even 1/3rd of the [untreated animals in a marmoset study Alexandros cites] died within the range of the study durations." On the more general argument: I have a higher credence of the effect modifier than he does. Perhaps the main thing I don't think he fully appreciates is just how few deaths need to be explained by this in order to substantially shift the RR. Even if this is the case for just a handful of control group deaths, the RR change is not trivial simply because of how low events the entire supposed benefit was in the first place. Furthermore, the new trials (ACTIV-6 400, ACTIV-6 600, COVID-OUT), while they all have very low mortality rates, all tip the needle in favor of the hypothesis. As expected, in the USA where the prevalence is near 0, all the deaths of those trials were in the ivermectin group (again, small event rates though). I could re-do the analysis with the new data even with his critiques of how he thinks it should be done (which is highly debatable) but I didn't end up doing it because at this point it would be an historical debate since the world has moved on from the topic. There are other points of course, there's quibbles line by line. I find Alexandros’ adjustment for Brazil somewhat convincing - not necessarily as a good adjustment, just in the sense that some adjustment needed to be done. I think the broader point is that results on the border of “statistical significance” often appear or go away depending on ambiguous decisions about coding single cases. Alexandros realizes this and includes a more gestalt style chart directly showing the correlation, which he says goes below the significance threshold when you recode the Brazilian studies. This chart seems to be missing some studies which might change its conclusions; it was made by a third party and Alexandros is going to get back to me with more information. Dr. Bitterman adds that more recent American studies strengthen his hypothesis. More discussion with Dr. Bitterman has also helped me better understand the context of this theory. Ivermectin does worst in studies of intermediate clinical endpoints: hospitalization, ICU admission, recovery time. It does best in studies of viral clearance rate and mortality. Viral clearance rate is a weak preclinical endpoint: not only is it especially susceptible to biases and file drawer effects, but it’s not that interesting unless it affects later clinical outcomes; many drugs change secondary endpoints but fail to change the things we care. Mortality is (usually) a strong and important endpoint; apparent positive results of ivermectin here require an explanation. The Strongyloides hypothesis tries to provide it. But I erred on my earlier post by holding it up as “the” explanation for a large and heterogenous group of studies which were mostly looking at endpoints other than mortality, or as a counter to ivmmeta’s analysis which found positive results everywhere for everything through statistical incompetence. I think I implicitly believed a stronger version of the worm hypothesis - that even in places without literal Strongyloides literally killing you, some people had some parasitic worms that were holding them back, ivermectin killed those worms, and that made them healthier overall and better able to deal with COVID. But nobody has asserted or defended that hypothesis and there’s no evidence for it. When I asked Dr. Bitterman, he pointed out that the opposite was at least as credible: parasitic worms depress the immune system, but immune overreaction is a major cause of death in COVID, so getting rid of them could make things worse rather than better. The original post should have explained this hypothesis better, devoted less emphasis to it, and focused more on publication bias and other issues that could explain the overall result. In some cases, these issues would have shed more light on the mortality statistics too. On my original post, I wrote: Parasitic worms are a significant confounder in some ivermectin studies, such that they made them get a positive result even when honest and methodologically sound: 50% confidence In retrospect this is framed too weakly - “significant” in “some” studies is compatible with irrelevant overall. Still, sticking to the spirit of what I meant, I think I would lower this guess to more like 35% now , and lower my overall estimate of how much of the mystery it explains even further. I’m not an expert on this, you shouldn’t care about my exact probability, and I’m only mentioning it to communicate clearly and try to hold myself accountable. V. Publication Bias Alexandros has various arguments against funnel plots in general, and Dr. Bitterman’s funnel plot in particular. Some of these arguments are reasonable, but taken together they would discredit 95 - 100% of all funnel plots everywhere. Trying to destroy the whole institution of funnel plots just because one of them disagrees with your hypothesis is . . . honestly a move I have to respect. I agree that these provide Bayesian evidence, rather than 100% irrefutable evidence, of publication bias, and need to be considered in the context of everything else going on. After doing that, I still think they’re publication bias. That makes publication bias more important. In the original post, I included this funnel plot from Dr. Bitterman: In case you haven’t seen one of these before: this plots how big an effect the study found (horizontal axis) against study size (vertical axis). Studies that find ivermectin had no effect are at the center (RR = 1), studies that find a strong curative effect are to the left, studies that find a strong harmful effect are to the right. When all studies are good, we have no reason to expect a correlation between study size and ivermectin efficacy - any deviations from the true effect should be random. This would look like a triangle centered around the true effect of the drug, with an equal number of studies on both sides. When there is a lot of publication bias, we should expect that small studies get published only if they find exciting results, and big studies get published regardless (because a lot of work went into them, someone will want to publish them, and journals will accept them regardless of how exciting they are). So here you would expect to see big studies around zero, and an asymmetric tail of smaller studies heading in the more-exciting direction. This is what we see on Dr. Bitterman’s plot, suggesting strong publication bias for ivermectin results. Alexandros’ full counterargument is here. Trying to sum it up: Funnel plots can sometimes look deceptive, or be misreported, and are generally suspect.
Inline links: 6, studies of intermediate clinical endpoints, https://substackcdn.com/image/fetch/$s_!rlgm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F174e569d-8a2c-4c11-a2ff-d355f4bce5b1_1200x808.jpeg, here
We got most of the studies listed in this analysis from ivmmeta, which is not a journal and tries to include all studies it knows about, including preprints. We have studies like Carvallo and Borody which are more like informal writeups about what happened than formal papers. “Publication bias” here would have to mean that someone hid the fact that their study happened at all - a stronger bar than just “a journal didn’t publish it”.
First, our selection of studies comes from IVMMeta, a site that included not just published articles but preprints and vaguely-written-up summaries of experimental results. This removes one source of publication bias: bias in what journals choose to publish. But it doesn’t remove another: bias in what studies get written up, even at the vague summary level. I asked Dr. Bitterman for his assessment of how often this happens; he says it’s very common, even with medium-sized studies. When I pressed him on how medium-sized, he says he knows institutions that might not publish a hundred-person RCT if its results were too boring. Most of the studies on IVMMeta were smaller than that, so publication bias is still likely.
Open Thread 264
February 20, 2023 · Original source
3: Alexandros responded by email to my ivermectin post. He wants to add that he talked to the person who made the strongyloides analysis graphic, who says all data points were in there but some are too small to see. He says the funnel plots I included are mislabeled and do not prove publication bias in every study on ivmmeta. And he continues to offer $25,000 to anyone who can get the TOGETHER study to release their data publicly, something which I agree all studies should either do or provide a justification for not doing.