A Novel Practical Approach for Directly Assessing COVID-19 Vaccine Efficacy against Hospitalization

We revisit a 2023 JAMA-published analysis of data from patients admitted for either COVID-19 or influenza to U.S. Veterans Administration (VA) hospitals in the fall/winter of 2022-2023. We note that baseline characteristics between the cohorts were similar, and the difference in vaccination rates was minimal, even after propensity score weighting. Had the vaccines been even minimally effective, significant differences in vaccination percentages would be expected. These findings call into question any efficacy claims associated with either the COVID-19 or influenza vaccines. Furthermore, the lack of efficacy of the influenza vaccines in reducing hospitalization is consistent with an earlier detailed study that found the same lack of efficacy, further validating the current result. This suggests that comparing vaccination rates between those hospitalized for different vaccine-preventable diseases can serve as a practical method for validating the vaccine efficacy claims of different vaccines.

Keywords:

Subject: Medicine and Pharmacology - Epidemiology and Infectious Diseases

Introduction

We recently became aware of a Research Letter by Xie et al. published in JAMA on April 6, 2023, in which the authors analyzed data from patients admitted to VA healthcare facilities between the dates of October 1, 2022, through January 31, 2023 for either influenza or COVID-19 [1]. Those patients diagnosed with both respiratory infections concurrently were excluded from the analysis.

Methods and Findings

Upon closer inspection of the data presented in this analysis, we noticed that the COVID-19 and influenza vaccination percentages of the two cohorts were nearly identical both before and after propensity score weighting. For example, after propensity score weighting for the boosted cohorts, these figures were 55.43% (1332 out of 2403) and 54.54% (4906 out of 8996) for influenza and COVID-19, respectively (see table in Xie et al.).

The Xie et al. table portrays the baseline characteristics between the two cohorts as being very similar with many characteristics having a standardized mean difference (SMD) of 0.1 or less. This similarity between the two cohorts was not unexpected, since both the influenza and SARS-CoV-2 viruses are highly contagious respiratory viruses that primarily infect the nasal cavity, pharynx and pulmonary system; and for both infectious diseases, the nearly identical cohorts have the same risk for hospitalization and death.

The similarity in the two cohorts in the Xie et al. study suggests that, for hospitalization stemming from these two respiratory infections, each cohort can serve as a control group against the other to accurately estimate vaccine efficacy against hospitalization. By utilizing each cohort as a control, we are able to avoid relying exclusively on potentially problematic mathematical models that purport to match the baseline characteristics of cases against a model control population, for which no individual-level data are available. Additionally, such comparators enable us to obtain a reasonably accurate estimate of vaccine efficacy against hospitalization.

Discussion and Conclusions

If ideal vaccines existed that prevented hospitalization entirely for each virus, one would expect the two cohorts to differ dramatically in the percent (%) vaccinated for the corresponding disease resulting in hospitalization. For example, for the VA patients included in the analysis who were hospitalized for COVID-19, it would be expected that 100% would not have received the COVID-19 vaccines, and that roughly 60% would have had the influenza vaccine. Likewise, for a perfect influenza vaccine, it would be expected that 100% of the patients hospitalized for the flu would be unvaccinated for influenza and that 80% would have received the COVID-19 vaccines based upon the VA data.

Given differences in the commonly claimed efficacy of the products and vaccination coverage in the populations, we expected to see significant differences in the vaccination percentages of the two groups in the Xie study. Yet, the opposite was true: there was little difference between the two cohorts as summarized in Table 1. In fact, the vaccination percentage difference between the two cohorts was among the lowest of all the characteristics measured; all standardized mean difference (SMD) values were under .05 (the actual values were .01, .02, .03, .04, and .05). After propensity score weighting, the SMD for the influenza vaccine dropped from .04 to .008. Even more troubling, of those who took the COVID-19 booster (which should have had the largest difference between the two cohorts if the vaccine performed as expected), actually had the smallest difference of any of the 21 baseline characteristics measured in the study: a SMD value of only .01.

Table 2 presents expected findings for hospitalization percentages based on the Xie et al. data if the influenza vaccine and COVID-19 injections conferred a 50% and 90% protection benefit, respectively, against hospitalization [2].

We are not aware of any viable alternative explanation for the observed data other than that neither vaccine provided any protection against hospitalization. This conclusion is consistent with previous research findings. For example, a carefully done study published in 2020 of 170 million cases and 7.6 million deaths showed that the influenza vaccine had no hospitalization or mortality benefit for the elderly [3]. In a recent review of multiple vaccine types, Morens et al., noted that “the rates of effectiveness of our best approved influenza vaccines would be inadequate for licensure for most other vaccine-preventable diseases” [4].

Wu et al. recently performed a meta-analysis of 68 studies (with data collected up to the end of 2022), revealing a marked decline in the effectiveness of the COVID-19 modified mRNA injections over time [5]. Taking into account publication bias, the lower bounds of confidence intervals suggest that effectiveness dropped down to near zero between 168 and 195 days of the injection. Notably, baseline effectiveness was further reduced after the emergence of Omicron in December 2021, falling below WHO’s “adequate vaccine response” criteria and indicating no benefit in terms of preventing Omicron variant infections, hospitalizations, or mortality. Two other published studies of subgroups of vaccinated hospitalized patients indicate that mortality rates may increase with additional COVID-19 vaccine doses [6,7]. Increased COVID-19 mortality among injected individuals is consistent with the phenomenon of pathogenic priming [8] and the likelihood that repeated injections result in immune dysfunction and increased mortality over time, as proposed by vaccinologist Geert Vanden Bossche [9].

We reached out to noted epidemiologist Ziyad Al-Aly, the paper’s senior author, but he was unable to explain how the vaccination percentages of the two cohorts could be so similar if either vaccine provided significant protection against hospitalization. The authors do acknowledge the potential influence of residual confounding; however, their propensity score weighting takes into account multiple covariates, including key comorbidities such as diabetes, cardiovascular disease, and hypertension.

Overall, the baseline data presented by Xie et al. suggest that neither the COVID-19 nor the influenza vaccines were effective in reducing the risk of hospitalization in late 2022 through early 2023. Had the immunizations been highly effective, significant differences in vaccination percentages would be expected between the two cohorts. Applying Occam’s razor, the most reasonable explanation is that neither vaccine conferred protection against severe illness and mortality in this context.

This method should not be used to directly assess mortality risk for two reasons: 1) patients are not hospitalized for mortality per se, and 2) patients can die outside of the hospital. With respect to a mortality benefit, we are unaware of any vaccine in the history of medicine that does not have a significant hospitalization benefit yet has a statistically significant mortality benefit. Thus, the lack of any significant reduction in hospitalization is strongly suggestive that both COVID-19 and influenza vaccines are unlikely to confer a mortality risk reduction for those diseases.

It appears that the Xie et al. study authors have inadvertently stumbled upon a new practical, interim method of directly estimating vaccine efficacy against hospitalization due to respiratory viruses by comparing the vaccination percentages in each cohort against each other. This approach bears some similarities to case-control vaccine effectiveness studies, in which the investigators match controls to cases (often via propensity scoring) or use the scores as a covariate weighting method [10]. Moreover, this method effectively obviates the “healthy vaccinee effect” that so often confounds other investigations of vaccine efficacy [11]. The general premise of the method seems sound, as it leverages real-world data to create a natural control group while minimizing biases such as the healthy vaccinee effect.

Since we observed minimal differences in vaccination percentages, this approach may offer a simpler and more expeditious way to assess vaccine efficacy of two or more vaccines without having to rely on complex models. Notably, however, the method assumes that the two cohorts are sufficiently comparable in terms of exposure and other risk factors. Though balancing techniques (such as inverse probability weighting) can be employed in other cases, this also comes with a set of assumptions that the underlying propensity scores subsume all relevant confounders. Therefore, while this screening approach can provide useful insights, it should be complemented by clinical trial data and other methods to ensure a comprehensive assessment of vaccine efficacy.

In conclusion, our assessment indicates that neither the influenza vaccine nor the COVID-19 vaccines provided any measurable difference in risk reduction of hospitalization for the very diseases they were designed to protect against. We propose that health authorities worldwide should reconsider the decision to approve both of these vaccines until such time as a real public health benefit can be demonstrated from the data using the method described in this paper, along with other complementary analytic methods. These findings reinforce recommendations to halt the global distribution of both the influenza vaccine and the COVID-19 vaccines [12].

Funding

No funding. The authors declare no conflict of interest.

Acknowledgments

We thank epidemiologist Kris Denhaerynck, PhD, from Basel, Switzerland, for insightful comments and confirmation of calculations.

Conflicts of Interest

None.

References

Xie, Y.; Choi, T.; Al-Aly, Z. Risk of Death in Patients Hospitalized for COVID-19 vs Seasonal Influenza in Fall-Winter 2022–2023. JAMA 2023, 329, 1697–1699. [Google Scholar] [CrossRef] [PubMed]
Centers for Disease Control and Prevention. Grading of Recommendations, Assessment, Development, and Evaluation (GRADE): Pfizer-BioNTech COVID-19 Vaccine. Available online: https://www.cdc.gov/vaccines/acip/recs/grade/covid-19-pfizer-biontech-vaccine.html (accessed on 31 July 2024).
Anderson, M.L.; Dobkin, C.; Gorry, D. The Effect of Influenza Vaccination for the Elderly on Hospitalization and Mortality: An Observational Study With a Regression Discontinuity Design. Ann. Intern. Med. 2020, 172, 445–452. [Google Scholar] [CrossRef] [PubMed]
Morens, D.M.; Taubenberger, J.K.; Fauci, A.S. Rethinking next-generation vaccines for coronaviruses, influenzaviruses, and other respiratory viruses. Cell Host Microbe 2023, 31, 146–157. [Google Scholar] [CrossRef] [PubMed]
Wu, N.; Joyal-Desmarais, K.; Ribeiro, P.A.B.; Vieira, A.M.; Stojanovic, J.; Sanuade, C.; Yip, D.; Bacon, S.L. Long-term effectiveness of COVID-19 vaccines against infections, hospitalisations, and mortality in adults: findings from a rapid living systematic evidence synthesis and meta-analysis up to December, 2022. Lancet Respir. Med. 2023, 11, 439–452. [Google Scholar] [CrossRef] [PubMed]
Adhikari, B.; Bednash, J.S.; Horowitz, J.C.; et al. Brief research report: impact of vaccination on antibody responses and mortality from severe COVID-19. Front. Immunol. 2024, 15, 1325243. [Google Scholar] [CrossRef]
Rzymski, P.; Pazgan-Simon, M.; Simon, K.; et al. Clinical Characteristics of Hospitalized COVID-19 Patients Who Received at Least One Dose of COVID-19 Vaccine. Vaccines 2021, 9, 781. [Google Scholar] [CrossRef] [PubMed]
Lyons-Weiler, J. Pathogenic priming likely contributes to serious and critical illness and mortality in COVID-19 via autoimmunity. J. Transl. Autoimmun. 2020, 3, 100051. [Google Scholar] [CrossRef] [PubMed]
Bossche, G.V. The Inescapable Immune Escape Pandemic. Pierucci Publishing, Aspen, CO. 2023. Available online: https://www.boswellbooks.com/book/9781956257809.
Verani, J.R.; Baqui, A.H.; Broome, C.V.; Cherian, T.; Cohen, C.; Farrar, J.L.; et al. Case-control vaccine effectiveness studies: Preparation, design, and enrollment of cases and controls. Vaccine 2017, 35, 3295–3302. [Google Scholar] [CrossRef] [PubMed]
Furst, T.; Straka, R.; Janosek, J. Healthy vaccinee effect: a bias not to be forgotten in observational studies on COVID-19 vaccine effectiveness. Pol. Arch. Intern. Med. 2024, 134, 16634. [Google Scholar] [CrossRef] [PubMed]
Mead, M.N.; Seneff, S.; Wolfinger, R.; Rose, J.; Denhaerynck, K.; Kirsch, S.; McCullough, P.A. COVID-19 Modified mRNA “Vaccines” Part 1: Lessons Learned from Clinical Trials, Mass Vaccination, and the Bio-Pharmaceutical Complex. Int. J. Vaccine Theory Pract. Res. 2024, 3, 1112–1178. [Google Scholar] [CrossRef]

Table 1. Baseline Vaccination Characteristics for COVID-19 and Influenza Cohorts.

	Hospitalized for influenza (n = 2403, %)	Hospitalized for COVID-19 (n=8996, %)	SMD
COVID-19 vaccine
Vaccinated	81.11	79.27	.05
Not vaccinated	18.89	20.73	.05
1 dose	4.74	4.27	.02
2 doses	21.51	20.46	.03
Boosted	54.85	54.54	.01
Influenza vaccine
Vaccinated	61.88	63.84	.04
Not vaccinated	38.12	36.16	.04

Table 1. These values are extracted from the table in the Xie paper showing the baseline vaccination characteristics reported for the two cohorts examined in the paper. Standardized Mean Difference (SMD) values of under 0.1 were considered to be well matched by the authors of the paper. The SMD values after propensity weighting were all under 0.1. The fact that the proportion vaccinated for both vaccines are nearly identical in the two columns indicates that there was no net protective effect against hospitalization for either vaccine.

Table 2. Expected Vaccination Rates for those Hospitalized for either Influenza or COVID-19.

	Hospitalized for influenza (n = 2403, %)	Hospitalized for COVID-19 (n=8996, %)
COVID-19 vaccine
Not vaccinated	18.89	69.96
1 dose	4.74	1.76
2 doses	21.51	7.97
Boosted	54.85	20.31
Influenza vaccine	46.89	63.84

Table 2. If the COVID-19 vaccines demonstrated a 90% reduction in hospitalization rates consistent with findings reported by the CDC [2], and the influenza vaccine showed a 50% reduction in hospitalization rates, we would have expected to see vaccination rates approximating those presented in this table for the two cohorts in the original study.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.

MDPI Initiatives

Important Links

Choose an area of interest and we will send you notifications of new preprints at your preferred frequency.

Disclaimer