Response to the comments to our recent preprint paper

Written with Frederik Schaltz-Buchholzer, Sebastian Nielsen and Peter Aaby

Since we put our paper “Randomised Clinical Trials of COVID-19 Vaccines: Do Adenovirus-Vector Vaccines Have Beneficial Non-Specific Effects?” online, we have received numerous comments. It was inevitable that the study would create discussion and sometimes also confusion and even fear. This is an aspect that is always part of our considerations when sharing results of studies of this character. Here, we attempt to reply to the comments. Many comments had overlapping themes; those we have gathered under a generic headline.

In brief, to examine the possible non-specific effects (NSEs) of the novel COVID-19 vaccines, we reviewed the randomised control trials (RCTs) of mRNA and adenovirus-vector COVID-19 vaccines reporting overall mortality, including COVID-19 deaths, accident deaths, cardiovascular deaths and other non-COVID-19 deaths. For overall mortality, with 74,193 participants and 61 deaths (mRNA:31; placebo:30), the mortality risk ratio (RR) for the two mRNA vaccines compared with placebo was 1.03 (95% CI=0.63-1.71). In the five adenovirus-vector vaccine trials there were 122,164 participants and 46 deaths (vaccine:16; controls:30). The RR for adenovirus-vector vaccines versus placebo/control vaccine was 0.37 (0.19-0.70). The adenovirus-vector vaccines were associated with protection against COVID-19 deaths (RR=0.11 (0.02-0.87)) and non-accident, non-COVID-19 deaths (RR=0.38 (0.17-0.88)). The two types of vaccines thus differed significantly with respect to impact on overall mortality as well as non-accident, non-COVID-19 deaths. We conclude that the data argue for performing RCTs of mRNA and adeno-vectored vaccines head-to-head comparing long-term effects on overall mortality.

We received the following generic comments:

1) The RCTs were not designed to test mortality, thus the data should not be tested for mortality.

Reply: It should be remembered that this is the only RCT data available to test the mortality effect of the vaccines. Should that analysis not be done because the RCTs were not designed to test mortality? The counterfactual underlying this comment seem to be that we should not investigate the mortality effect of the new vaccines, given that no trial was designed to test mortality effects. In contrast to this viewpoint, we believe that we should get the most out of the best data we have.

2) There were too few deaths in these RCTs for such analyses to be meaningful.

Reply: This criticism of too few deaths in the trials is a criticism of the basis on which the COVID-19 vaccines have been approved and disseminated to the entire global population. We are the first to lament that the RCTs were not larger, and that the follow-up was not continued longer (controls were offered vaccination after a few months), because this limits our ability to provide more certain estimates of the long-term effects of the different vaccines on all-cause mortality. However, as pointed out above: this is the best data we have. The available data does suggest that there could be major differences in mortality effects between the two types of vaccines. This, of course, needs to be investigated further.

3) The Johnson&Johnson (J&J) trial carried the results for the adenovirus vector analysis; if it was pulled out, the adenovirus vector results are no longer statistically significant.

Reply: It is always possible to sequentially chop a data set into smaller pieces until effects are no longer statistically significant.

4) There is a more updated follow-up of the J&J trial published in late March 2022, if that is included, the adenovirus vector results would no longer be statistically significant.

Reply:  As indicated in our paper, our analysis was based on a search in January 2022, and included the J&J data published in December 2021. It therefore did not include the subsequent J&J paper. The paper from March 2022 does not have information on cause of death, apart from death from COVID-19. We have written to the authors of the March 2022 paper; they have not yet responded. We will update the paper once they respond.

The March 2022 paper has information on all-cause mortality. There were 28 deaths in the vaccine group vs. 55 among controls. We have therefore calculated the adenovirus vector effect on all-cause mortality including the updated follow-up. The combined RR for overall mortality in the adenovirus-vector RCTs is 0.52 (0.35-0.77), a significant difference from the null finding in the mRNA RCTs (p=0.035). Hence, with respect to all-cause mortality (the outcome that we can assess using the March data), inclusion of the updated J&J data does not change the main conclusions that adenovirus vector vaccines are associated with significant reductions in all-cause mortality and this effect was statistically different from the effect of the mRNA vaccines.

5) There was only a statistically significant reduction in mortality with J&J vaccine, not with AstraZeneca. This undermines the hypothesis that the adenovirus platform in itself has a non-specific protective effect on the overall mortality.

Reply: We have not claimed that AstraZeneca had a significant effect, but current figures point to a 48% reduction (95% CI=-18 to 77%). Hence, the effect points in the same direction as that seen for the J&J vaccine.

6) There was so much COVID-19 mortality during the J&J trial that most deaths were presumably due to COVID-19, and therefore the vaccine provided such beneficial estimates.

Reply: The J&J trial had COVID-19 as outcome and COVID-19 was ruled out by the authors as the cause of death for most deaths (https://nejm.org/doi/suppl/10.1056/NEJMoa2101544/suppl_file/nejmoa2101544_appendix.pdf (Page 48)).

7) There could be uncertainty regarding the causes of death.

Reply: Regrettably causes of death were not reported in detail. Apart from COVID-19, which was the main outcome for the trials, and that we therefore assume was diagnosed fairly accurately, and accidents, that are also rather clear-cut (though one could argue that a car accident could happen as a result of a heart attack), we certainly acknowledge in our preprint that there could be uncertainties in the causes of death. This is the reason why we focus on all-cause mortality and “non-COVID-non-accident mortality”.

8) There may be differences in the populations in the different RCTs so factors other than the vaccine may be the reason for the difference in mortality effects between the two vaccine types. These can be, for example, gender, age composition, state of health and other exposures.

Reply: This is a relevant limitation that we also discuss in our preprint. Certainly, differences in the populations included in the RCTs may have influenced the comparison of mRNA and adenovirus vector effect estimates. If, for example, within the adenovirus vector RCTs there was more COVID-19, or greater mortality from COVID-19, than in the mRNA vaccine RCTs, this would favour the adenovirus vector vaccines. However, there should be very substantial differences between the different RCTs to arrive at an overall mortality reduction of 48% (updated with the newest J&J data) for adenovirus vector vaccines and a complete absence of mortality effect of -3% for mRNA vaccines.

Noteworthy, there is some overlap between the countries in which the vaccines were tested, and in the most important RCTs that contribute deaths, the average age was 50 years (AstraZeneca), 51 years (Pfizer), 51 years (Moderna) and 52 years (Johnson & Johnson), respectively.

9) The risk of dying from heart disease increases with age. This means that the difference in mortality from the different vaccines may be due to age, not the vaccines.

Reply: This argument is not correct. Within the RCTs, vaccination is compared with placebo. Age and other background factors were therefore evenly distributed between the randomisation groups per trial design. Thus, there is no difference in the age of the vaccinated and controls within the mRNA studies and within the adenovirus studies. Nonetheless, in the mRNA studies, there is a trend for higher risk of cardiovascular death for the vaccinated vs. controls whereas in the adenovirus studies, there is significantly lower cardiovascular mortality in the vaccinated vs. controls.

10) One can always tease out whatever one wants to see in such reanalysis of the data.

Reply: We do not understand this comment. In an analysis that is based on already published data, and where all relevant studies are included, one cannot manipulate data to get the results one wants. It should be noted that for many years, our main research agenda has been to examine the effect of health interventions on the overall health of the population, so the systematic approach applied in this study is not at all new. We have found that live vaccines, such as BCG vaccine, measles-containing vaccines and oral polio vaccine, have beneficial non-specific effects on overall health. Adenovirus vector vaccines are different from traditional live vaccines in that they are not replicating. Therefore, it was not our a priori hypothesis that they would be associated with a lower mortality rate. mRNA vaccines are a new vaccine type for which we also had no a priori expectations. We simply did what we have done numerous times before for other vaccines: studied the effect on all-cause mortality. We have presented our methods in the manuscript, so that other researchers can reproduce the results, but the data was already published, and the analysis offers very limited choices by the investigators, so it is really not possible to manipulate the data or the analysis.

11)   The authors did not write a protocol to describe their analysis before they did it.

Reply: We refer to the answer to the comment above. The study was explorative and since we did not know how much information we could get from the authors it would have little meaning to beforehand limit the kind of information we would collect and analyse.

***

In summary, above, we have attempted to respond to the comments we have received – both the more and the less relevant. We appreciate the scientific debate about our study and the attention it has received. We will update this article if we receive other comments.

We reflect on the fact that the comments primarily seem to dismiss results because the RCTs were not designed to test mortality and there were too few deaths. We would have hoped that more readers would have expressed curiosity and asked themselves and us: If this is true then how can we investigate it further? That is the important and fruitful question.

Our key message is that yes, data on the overall mortality effect of the new vaccines is regrettably limited, but nonetheless it does show differences in mortality effect between the two major vaccine types that if true would have major global health implications. This should prompt immediate action in terms of more research.

We are happy to continue the discussions of these first findings in relation to the overall health effects of the new vaccines.