Astral Codex Ten

TOGETHER

16 min read

TOGETHER

Article

TOGETHER is a recurring organization in the Astral Codex Ten archive, appearing 2 times across 2 issues between October 09, 2022 and February 01, 2023. The archive places it in contexts such as “the big COVID drug trials (TOGETHER, ACTIV-6, COVID-OUT)”; “TOGETHER is missing ‘time since symptom onset’ for 23% of its patients”. It most often appears alongside Alexandros Marinos, 2006 Ioannidis paper, ACTIV-6.

Metadata

  • Category: Organizations
  • Mention count: 2
  • Issue count: 2
  • First seen: October 09, 2022
  • Last seen: February 01, 2023

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Open Thread 245
October 09, 2022 · Original source
3: Alexandros Marinos continues his critique of my ivermectin post, and his broader ivermectin advocacy. I can’t remember if I’ve said this before, but I commit to writing a summary and response within four months of him being done, which as far as I can tell is not yet (yes, four months is a long time, but it’s a long series and I’m really busy this winter). His most recent post argues that the big COVID drug trials (TOGETHER, ACTIV-6, COVID-OUT) haven’t made their data public, and offers to donate $25,000 to a charity of my choice if I can get them to do so. I have no idea how to do this, but I agree that they should; if anyone from these trials wants to get in touch with me and talk about it and I would be interested in hearing what’s going on.
Inline links: continues, my ivermectin post
Response To Alexandros Contra Me On Ivermectin
February 01, 2023 · Original source
Alexandros doesn’t dispute that one of Cadegiani’s trial had some impossible-seeming statistics, but says we shouldn’t jump to allegations of fraud, shouldn’t let this unduly influence our opinion of Cadegiani’s other trials, and also accuses Kyle Sheldrick, the person who discovered the discrepancy, of doing other bad things. My responses: Alexandros’ Point 1 is fair-ish. Since this person appears to be commiting pretty substantial fraud and doing some strange things, I thought it was useful to highlight the ways in which he is weird and suspicious, rather than the ways he is prestigious and impressive. But probably I went too far in this. His Point 2/3 is completely fair, and I’m sorry for getting this wrong. I may have unthinkingly copied it from forbetterscience.com, which made this mistake before me, or I might have just failed at reading comprehension on this translated Portugese-language article I linked. In either case, I apologize to Cadegiani. This is already on my Mistakes page as of June 2022 when Alexandros wrote his original article. His Point 4 is correct, although based on information that came out after I wrote my article. All that was available in English when I wrote was that the Brazilian government was considering accusing Cadegiani of crimes against humanity. I think I did an okay job noting that I was guessing at their reasoning (rather than reporting a known fact), and as written I did make clear that I thought he was innocent of the specific charge. Still, I appreciate the clarification. His Point 5 is - I do feel like Alexandros is having a sort of missing mood on the fact that one of Cadegiani’s big pro-ivermectin studies contains impossible data. While this is not proof of fraud or incompetence, it is some Bayesian evidence for both. And while fraud or incompetence in one of your studies supporting ivermectin is not proof that your other studies supporting ivermectin are also fraudulent/incompetent, it is, again, Bayesian evidence. Alexandros makes a big deal of there being four corrections in the BMJ article attacking Cadegiani, as if now the BMJ has admitted they were wrong all along, whereas these were mostly on unrelated details and the BMJ definitely did not correct the quotes about how his study was “an ethical cesspool of violations” or how “in the entire history of the National Health Council, there has never been such disrespect for ethical standards and research participants in the country”1. I feel like if his Science Olympiad medals are an important part of the story, these kinds of things are an important part too. Still, several of Alexandros’ points were entirely correct, and I appreciate the corrections. Babalola et al (still disagree with Alexandros) OE Babalola (I incorrectly wrote this name as “Babaloba” in the original) did a Nigerian study which found that ivermectin decreased the amount of time it took before people tested negative for COVID. I described this study as: This was a Nigerian RCT comparing 21 patients on low-dose ivermectin, 21 patients on high-dose ivermectin, and 20 patients on a combination of lopinavir and ritonavir, a combination antiviral which later studies found not to work for COVID and which might as well be considered a placebo. Primary outcome, as usual, was days until a negative PCR test. High dose ivermectin was 4.65 days, low dose was 6 days, control was 9.15, p = 0.035. Gideon Meyerowitz-Katz, part of the team that detects fraud in ivermectin papers, is not a fan of this one. He doesn’t say there what means, but elsewhere he tweets [this figure highlighting how the study has “Numerous impossible numbers”] I think his point is that if you have 21 people, it’s impossible to have 50% of them have headache, because that would be 10.5. If 10 people have a headache, it would be 47.6%; if 11, 52%. So something is clearly wrong here. Seems like a relatively minor mistake, and Meyerowitz-Katz stops short of calling fraud, but it’s not a good look. I’m going to be slightly uncomfortable with this study without rejecting it entirely, and move on. Alexandros calls this The Sullying Of Babalola Et Al, and says I “followed Gideon Meyerowitz-Katz off a cliff” by unfairly “lambasting” the innocent Babalola. I “[made] a mountain out of a molehill”. Alexandros quotes a commenter who found that the most likely explanation for the “impossible numbers” in Babaloba was missing data, and notes that usually-anti-ivermectin researcher Kyle Sheldrick had evaluated the raw data and found no fraud. Alexandros concludes: As far as I can tell, Scott discarded a good study here, and besmirched the reputation of the researchers by amplifying flimsy allegations that were known to be off-base at the time that the article was written. I don’t think I did anything especially wrong here. There was a chart that didn’t make sense. It turned out not to make sense because some data was missing. I said “[this] seems like a relatively minor mistake, and Meyerowitz-Katz stops short of calling fraud, but it’s not a good look. I’m going to be slightly uncomfortable with this study without rejecting it entirely, and move on.” I was right that it was a minor mistake, I was right that it wasn’t fraud, and I was right not to reject the study. I didn’t have the exact explanation (missing data), so I did not mention it, but I think I made the correct guess about the sort of explanation it was. I don’t understand why Alexandros acts like I said the study wasn’t worth keeping, or that there was no innocent explanation, or that I was accusing the researchers of fraud, when in fact I said the opposite of all those things, pretty explicitly.2 Carvallo et al (Alexandros 25% right) This was an Argentine study. I described it as: This one has all the disadvantages of Espitia-Hernandez, plus it’s completely unreadable. It’s hard to figure out how many patients there were, whether it was an RCT or not, etc. It looks like maybe there were 42 experimentals and 14 controls, and the controls were about 10x more likely to die than the experimentals. Seems pretty bad. On the other hand, another Carvallo paper was retracted because of fraud: apparently the hospital where the study supposedly took place said it never happened there. I can’t tell if this is a different version of that study, a pilot study for that study, or a different study by the same guy. Anyway, it’s too confusing to interpret, shows implausible results, and is by a known fraudster, so I feel okay about ignoring this one. Alexandros responds here. Attempting to summarize his points: He agrees this study is extremely confusing.
Inline links: forbetterscience.com, this translated Portugese-language article, 1, not a fan of this one, this figure, The Sullying Of Babalola Et Al, 2, another Carvallo paper was retracted, here
The control groups in high-worm-prevalence-area studies had no more deaths than in low-worm-prevalence-area studies. If the worms were killing people in the control groups (who ivermectin was then saving in the treatment groups) you would expect more deaths. You can find arguments for all these points at the link. (One additional thing Alexandros does that I really like: he compares the Strongyloides hypothesis - as an attempt to explain why these studies keep getting such different results - to other hypotheses. For example, studies in Latin America get negative results more often than others. This really feels like confronting the real question. He finds that Latin American studies do find lower efficacy for ivermectin than the other mostly Asian studies, and hypothesizes that this is because ivermectin is very popular in Latin America, the “control” group illicitly takes it without telling the researchers, and so these studies are inadvertantly comparing two ivermectin groups. This is another clever and elegant theory. Unfortunately, the recent spate of negative American studies sink it6. Still, I agree there is a strong geographic element here; worms are one possible explanation, but there are others - including the scientific culture in different countries. I appreciate Alexandros highlighting how much this is true.) I asked Dr. Bitterman for his thoughts. He reiterates that although steroids are one major cause of Strongyloides hyperinfection, another is eosinopenia, a decrease in the immune cells that fight parasites. COVID can cause eosinopenia directly, so just because a COVID patient didn’t get steroids, or was only on steroids for a short period, doesn’t prove that the patient couldn’t have had hyperinfection. On the mixing of different sources to get Strongyloides prevalence data, he said: As mentioned in the paper, when available we attempted to granulate by regional prevalence. This was often not possible because robust data did not available for a given country. Brazil is a large country (and multiple different studies in our analysis were in Brazil) with variability and Paula was a robust study. We decided to attempt to granulate instead of stacking the Brazil countries with the same prevalence even though they are in very different regions. His re-analysis is a crude one, and he often switches between using that analysis and the ecological model study. At the time of out paper's publication, the ecological model was not available. I offered to re-do the whole analysis with that study's raw data with a mutually agreed upon methodology, which we have not fully ironed down yet. On the long delay before hyperinfection kills: I don't think it happens in 1 or 2 weeks. But 3-4 weeks (within almost all study durations) is certainly not unheard of (again, without being treated). Even 1/3rd of the [untreated animals in a marmoset study Alexandros cites] died within the range of the study durations." On the more general argument: I have a higher credence of the effect modifier than he does. Perhaps the main thing I don't think he fully appreciates is just how few deaths need to be explained by this in order to substantially shift the RR. Even if this is the case for just a handful of control group deaths, the RR change is not trivial simply because of how low events the entire supposed benefit was in the first place. Furthermore, the new trials (ACTIV-6 400, ACTIV-6 600, COVID-OUT), while they all have very low mortality rates, all tip the needle in favor of the hypothesis. As expected, in the USA where the prevalence is near 0, all the deaths of those trials were in the ivermectin group (again, small event rates though). I could re-do the analysis with the new data even with his critiques of how he thinks it should be done (which is highly debatable) but I didn't end up doing it because at this point it would be an historical debate since the world has moved on from the topic. There are other points of course, there's quibbles line by line. I find Alexandros’ adjustment for Brazil somewhat convincing - not necessarily as a good adjustment, just in the sense that some adjustment needed to be done. I think the broader point is that results on the border of “statistical significance” often appear or go away depending on ambiguous decisions about coding single cases. Alexandros realizes this and includes a more gestalt style chart directly showing the correlation, which he says goes below the significance threshold when you recode the Brazilian studies. This chart seems to be missing some studies which might change its conclusions; it was made by a third party and Alexandros is going to get back to me with more information. Dr. Bitterman adds that more recent American studies strengthen his hypothesis. More discussion with Dr. Bitterman has also helped me better understand the context of this theory. Ivermectin does worst in studies of intermediate clinical endpoints: hospitalization, ICU admission, recovery time. It does best in studies of viral clearance rate and mortality. Viral clearance rate is a weak preclinical endpoint: not only is it especially susceptible to biases and file drawer effects, but it’s not that interesting unless it affects later clinical outcomes; many drugs change secondary endpoints but fail to change the things we care. Mortality is (usually) a strong and important endpoint; apparent positive results of ivermectin here require an explanation. The Strongyloides hypothesis tries to provide it. But I erred on my earlier post by holding it up as “the” explanation for a large and heterogenous group of studies which were mostly looking at endpoints other than mortality, or as a counter to ivmmeta’s analysis which found positive results everywhere for everything through statistical incompetence. I think I implicitly believed a stronger version of the worm hypothesis - that even in places without literal Strongyloides literally killing you, some people had some parasitic worms that were holding them back, ivermectin killed those worms, and that made them healthier overall and better able to deal with COVID. But nobody has asserted or defended that hypothesis and there’s no evidence for it. When I asked Dr. Bitterman, he pointed out that the opposite was at least as credible: parasitic worms depress the immune system, but immune overreaction is a major cause of death in COVID, so getting rid of them could make things worse rather than better. The original post should have explained this hypothesis better, devoted less emphasis to it, and focused more on publication bias and other issues that could explain the overall result. In some cases, these issues would have shed more light on the mortality statistics too. On my original post, I wrote: Parasitic worms are a significant confounder in some ivermectin studies, such that they made them get a positive result even when honest and methodologically sound: 50% confidence In retrospect this is framed too weakly - “significant” in “some” studies is compatible with irrelevant overall. Still, sticking to the spirit of what I meant, I think I would lower this guess to more like 35% now , and lower my overall estimate of how much of the mystery it explains even further. I’m not an expert on this, you shouldn’t care about my exact probability, and I’m only mentioning it to communicate clearly and try to hold myself accountable. V. Publication Bias Alexandros has various arguments against funnel plots in general, and Dr. Bitterman’s funnel plot in particular. Some of these arguments are reasonable, but taken together they would discredit 95 - 100% of all funnel plots everywhere. Trying to destroy the whole institution of funnel plots just because one of them disagrees with your hypothesis is . . . honestly a move I have to respect. I agree that these provide Bayesian evidence, rather than 100% irrefutable evidence, of publication bias, and need to be considered in the context of everything else going on. After doing that, I still think they’re publication bias. That makes publication bias more important. In the original post, I included this funnel plot from Dr. Bitterman: In case you haven’t seen one of these before: this plots how big an effect the study found (horizontal axis) against study size (vertical axis). Studies that find ivermectin had no effect are at the center (RR = 1), studies that find a strong curative effect are to the left, studies that find a strong harmful effect are to the right. When all studies are good, we have no reason to expect a correlation between study size and ivermectin efficacy - any deviations from the true effect should be random. This would look like a triangle centered around the true effect of the drug, with an equal number of studies on both sides. When there is a lot of publication bias, we should expect that small studies get published only if they find exciting results, and big studies get published regardless (because a lot of work went into them, someone will want to publish them, and journals will accept them regardless of how exciting they are). So here you would expect to see big studies around zero, and an asymmetric tail of smaller studies heading in the more-exciting direction. This is what we see on Dr. Bitterman’s plot, suggesting strong publication bias for ivermectin results. Alexandros’ full counterargument is here. Trying to sum it up: Funnel plots can sometimes look deceptive, or be misreported, and are generally suspect.
Inline links: 6, studies of intermediate clinical endpoints, https://substackcdn.com/image/fetch/$s_!rlgm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F174e569d-8a2c-4c11-a2ff-d355f4bce5b1_1200x808.jpeg, here
You can read Alexandros’ criticisms of ACTIV-6 trial here (1, 2, 3, 4), and his criticisms of I-TECH here. I don’t think he’s criticized COVID-OUT yet, but I’m sure it’s only a matter of time.
Inline links: 1, 2, 3, 4, here