Very interesting! Thanks for posting this!

The non-equivalence of statistical significance and clinical importance has long been recognized, but this error of interpretation remains common. Although a significant result in a large study may sometimes not be clinically important, a far greater problem arises from misinterpretation of non-significant findings. By convention a P value greater than 5% (P0.05) is called "not significant." Randomized controlled clinical trials that do not show a significant difference between the treatments being compared are often called "negative." This term wrongly implies that the study has shown that there is no difference, whereas usually all that has been shown is an absence of evidence of a difference. These are quite different statements.

This is very pertinent to ivermectin. There are a handful of trials that failed to show statsig results. All but 1 or 2, though, had results in the right direction. Ivermectin was beneficial, just not statsig beneficial due to small N + insufficient effect size.

The sample size of controlled trials is generally inadequate, with a consequent lack of power to detect real, and clinically worthwhile, differences in treatment. Freiman et al1 found that only 30% of a sample of 71 trials published in the New England Journal of Medicine in 1978-9 with P0.1 were large enough to have a 90% chance of detecting even a 50% difference in the effectiveness of the treatments being compared, and they found no improvement in a similar sample of trials published in 1988. To interpret all these "negative" trials as providing evidence of the ineffectiveness of new treatments is clearly wrong and foolhardy. The term "negative" should not be used in this context.

It's not a negative results, it's a not-statsig result. Big difference.

A recent example is given by a trial comparing octreotide and sclerotherapy in patients with variceal bleeding. The study was carried out on a sample of only 100 despite a reported calculation that suggested that 1800 patients were needed. This trial had only a 5% chance of getting a statistically significant result if the stated clinically worthwhile treatment difference truly existed. One consequence of such low statistical power was a wide confidence interval for the treatment difference. The authors concluded that the two treatments were equally effective despite a 95% confidence interval that included differences between the cure rates of the two treatments of up to 20 percentage points.

Similar evidence of the dangers of misinterpretation of non-significant results is found in numerous meta-analyses (overviews) of published trials, when few or none of the individual trials were statistically large enough. A dramatic example is provided by the overview of clinical trials evaluating fibrinolytic treatment (mostly streptokinase) for preventing reinfarction after acute myocardial infarction. The overview of randomized controlled trials found a modest but clinically worthwhile (and highly significant) reduction in mortality of 22%, 4 but only five of the 24 trials had shown a statistically significant effect with P<0.05. The lack of statistical significance of most of the individual trials led to a long delay before the true value of streptokinase was appreciated.

LOL / Grrrr. That's the point of meta-analysis: Group the N to increase the statistical power!

While it is usually reasonable not to accept a new treatment unless there is positive evidence in its favor, when issues of public health are concerned we must question whether the absence of evidence is a valid enough justification for inaction. A recent publicized example is the suggested link between some sudden infant deaths and antimony in cot mattresses. Statements about the absence of evidence are common—for example, in relation to the possible link between violent behavior and exposure to violence on television and video, the possible harmful effects of pesticide residues in drinking water, the possible link between electromagnetic fields and leukemia, and the possible transmission of bovine spongiform encephalopathy from cows. Can we be comfortable that the absence of clear evidence in such cases means that there is no risk or only a negligible one?

When we are told that "there is no evidence that A causes B" we should first ask whether absence of evidence means simply that there is no information at all. If there are data we should look for quantification of the association rather than just a P value. Where risks are small P values may well mislead: confidence intervals are likely to be wide, indicating considerable uncertainty. While we can never prove the absence of a relation, when necessary we should seek evidence against the link between A and B—for example, from case-control studies. The importance of carrying out such studies will relate to the seriousness of the postulated effect and how widespread is the exposure in the population.

The 2 case-control studies I'm aware of (N=3 and N=100) were both positive for ivermectin.

permalink

When they started using Ivermectin to kill the parasites that cause River Blindness, it caused some itching. Not because of the medicine, but because the worms were dying and their death throes caused that sensation. The medicine they used before killed the worms too quickly and clumps of dead worms would cause bad side effects. Ivermectin was perfect...not too fast, not too slow. They still show itching as a possible side effect even though it's not directly because of Ivermectin. I don't know why this article reminded me of that, just thought I'd share.

permalink

https://www.bmj.com/content/311/7003/485 Statistics notes: Absence of evidence is not evidence of absence

BMJ 1995; 311 doi: https://doi.org/10.1136/bmj.311.7003.485 (Published 19 August 1995)Cite this as: BMJ 1995;311:485

Good reference to quote - says in a concise way what common sense suggests when one sees headlines talking about "negative" trial results - when all it is is a failure to show positive results, which can also happen if the study is not powered enough (ie not had sufficient subjects to eke out a sensible result).

Often times attributed to a weakness in the design of the study. Which suggests that the designers of studies would know of or have a rough idea (from earlier smaller studies) of the scale of effect to expect, and would have designed the study to be large enough to see a visible difference in that metric.

For example, a small study on impact on mortality of ivermectin may not show a result because with let's say 50 people in placebo and ivermectin arm each, there are not expected to be many deaths (if overall death rate from covid19 is 2.5 percent).

However, even a 50 person arm can show large differences between placebo and drug arm, if you look for an effect which would be apparent in the placebo arm - for example, hospitalizations, or need for oxygen. Or duration of symptoms.

A 50 patient per arm study would be considered "underpowered" for answering questions on mortality if patients are mild and moderate patients ie full spectrum of patients. Because not enough deaths will occur in even the placebo arm. In order to get numbers which could be compared across placebo vs drug arm, you would need 10 or 20 deaths in the placebo arm, in order to see a strong reduction in the drug arm down to 5 or 10 deaths. Which would mean having 500-1000 or so patients in each arm (if death rate is 2 percent).

But 50 patients may be enough to see differences in mortality between placebo and drug arm if they are all ICU patients - which may have 25 percent mortality rate in ICU patients. But again you will have difficulty matching the initial disease severity in the placebo vs drug arm with so few patients (harder to exclude patients - because then have even fewer patients left after pruning the groups to match disease severity).

But 50 patients may be sufficient to demonstrate improvement in drug arm, if you are looking at hospitalizations, or need for oxygen, if for example zero needed oxygen or hospitalization in the drug/treatment arm, while 25 needed that in the placebo arm.

permalink