The ubiquitous use of COVID-19 vaccination during the pandemic makes it challenging to quantify their effect. While comparisons can be made along the time (comparing outcomes by vaccination rate), most estimates rely on modeling using vaccine efficacy taken from clinical trials. Significantly, estimates from averted mortality (AM) models impact policy decisions, and their assumptions must be transparent and replicable.
To assesses the accuracy of model assumptions for estimates of AM due to COVID-19 vaccines. Recognizing the need for simplifications and assumptions in model building, the research seeks areas for improvement in current methodologies.
The study employs a thorough analysis of existing models that quantify the impact of mass vaccination on AM, both globally and in specific countries/regions. The research scrutinizes the assumptions made by these models and identifies areas where they might overstate the degree of AM due to vaccination. This study also makes inter-model comparisons to find outlier models.
Several assumptions in existing models tend to overstate the level of AM from COVID-19 vaccines significantly. This correlation raises questions about the accuracy of estimates regarding positive AM due to mass vaccination. This investigation finds a notable outlier for AM modeling, a Canadian study.
We highlights the need for improved epidemiological modeling in assessing the impact of vaccination. Assumptions tend to overstate AM, motivating the importance of grounding public health responses to infectious diseases in robust and rigorous analysis. The research contributes to refining our understanding of the consequences of mass vaccination during the COVID-19 pandemic and encourages a more nuanced approach to policy decisions.
La vacunación masiva contra el COVID-19 durante la pandemia hace que sea un reto cuantificar su efecto. Aunque se pueden hacer comparaciones a lo largo del tiempo (comparando resultados según la tasa de vacunación), la mayoría de las estimaciones se basan en modelos que utilizan la eficacia de la vacuna tomados de ensayos clínicos. Las estimaciones de los modelos de mortalidad evitada (ME) repercutan en las decisiones políticas, por lo que sus supuestos deben ser transparentes y reproducibles.
Evaluar los supuestos de los modelos para las estimaciones de ME por vacunas COVID-19. Reconociendo la necesidad de simplificaciones y suposiciones en la construcción de modelos.
El estudio analiza algunos modelos existentes que cuantificaron el impacto de la vacunación masiva en la ME, tanto a nivel mundial como en países/regiones específicos. Se examinaron los supuestos de estos modelos e identificaron áreas en las que podrían exagerar la ME. Este estudio también realizó comparaciones entre modelos para encontrar modelos atípicos.
Varios supuestos de los modelos existentes tienden a exagerar significativamente el nivel de ME por las vacunas COVID-19. Esta correlación plantea interrogantes sobre la exactitud de las estimaciones relativas a la ME positiva debida a la vacunación masiva. Esta investigación encontró un notable caso atípico en la modelización de la ME: estudio canadiense.
Se destaca la necesidad de mejorar la modelización epidemiológica para evaluar el impacto de la vacunación. Los supuestos tienden a exagerar la ME, lo que motiva la importancia de fundamentar las respuestas de salud pública a las enfermedades infecciosas en análisis sólidos y rigurosos. La investigación contribuye a refinar nuestra comprensión de las consecuencias de la vacunación masiva durante la pandemia de COVID-19 y fomenta un enfoque más matizado de las decisiones políticas.
Preprint: This article was previously published as a preprint on Preprint.org, available at:
Objective
Assess validity of averted mortality modeling assumptions
Study design
Literature search for any estimates for averted mortality due to Covid-19 vaccines
Source of information
Peer-reviewed articles, preprints, institutional and governmental reports modeling Covid-19 vaccine averted mortality
Population / sample
Averted mortality modelling papers estimating number of deaths averted due to Covid-19 vaccines
Statistical analysis
Comparison of modeling assumptions between different models. Comparison of actual Covid-19 mortality and estimated numbers of averted deaths to compare models
Main findings
Most models have a roughly 1:1 ratio of Covid-19 deaths and estimated averted deaths. The results of a Canadian study estimate far more deaths averted. Several models are based on assumptions which will tend to overstate averted mortality due to Covid-19 vaccines.
Averted mortality estimates from the public health response to COVID-19 have been used as justification for public health measures, including masking, lockdowns and vaccination. Public health policies operate under the assumption that the putative positive impacts of health measures (averted mortality), more than balance out their costs. Lockdowns
Claims that the Covid-19 vaccines saved lives are central to the policy of providing and/or mandating vaccines. However, claims that vaccines averted mortality are difficult to verify, relying on models. National comparisons of vaccination rate with excess mortality demonstrate a statistically significant positive correlation between vaccination uptake and 2022 excess deaths
At a subcommittee hearing on oversight of the US Centers for Disease Control (CDC) policies and decisions during the Covid-19 pandemic, CDC director Rochelle Walensky cited a study by the Commonwealth fund, a US think tank, claiming that the vaccines had saved 3.2 million lives in the US, prevented 18.5 million hospitalizations and saved $1.15 trillion in health care costs
While simplifications and assumptions are essential to model building, this work assesses the regularity of certain assumptions and their possible effect on the estimated value. In examination of the averted mortality models for Covid-19 vaccination, we search for areas of improvement, while understanding that simplifications are inherent in model building.This article aims to improve modeling and to ensure that uncertainties are communicated clearly.
The aim was to identify, assess, and compare modeling studies that estimated the number of deaths averted due to COVID-19 vaccination, with a focus on examining the methodological assumptions and transparency of these models.
We included studies that:
Modeled the estimated number of COVID-19-related deaths averted due to vaccination campaigns. Were based on national or regional datasets which would correspond to a known population (e.g., USA, Canada, Israel, etc.). Were published as peer-reviewed articles, preprints, institutional reports, or government documents between December 2020 and April 2023.
Exclusion criteria:
Studies that only estimated vaccine effectiveness without linking to mortality averted. Studies that provide estimates of averted mortality for factors not related to vaccination, such as lockdowns or mask mandates.
A systematic search was conducted across the following sources:
Google Scholar Reference lists of relevant reviews and reports were also screened manually.
Keywords included: "COVID-19 vaccine", "averted deaths", "mortality modeling", "vaccine efficacy", "counterfactual analysis", and "public health impact".
Titles and abstracts were independently screened by two reviewers to assess eligibility. Full-text articles were retrieved for all potentially relevant studies. Disagreements were resolved through discussion or consultation with a third reviewer.
A total of 12 studies were included after screening for inclusion criteria and relevance (9-20).
We developed a standardized data extraction form to collect:
Study region and time frame Reported COVID-19 deaths during the modeling period Averted mortality estimates (absolute and 95% CI) Assumptions regarding vaccine efficacy against death (VE<sub>death</sub>) Whether models accounted for waning immunity Treatment of vaccine adverse events Availability of code and model parameters Model type (e.g., regression, compartmental, counterfactual)
Each study was reviewed to extract the above variables, and results were tabulated.
We perform a literature search for estimates of deaths averted due to Covid-19 vaccines to compare models. Our search finds eleven articles providing an estimate of deaths averted due to Covid-19 vaccines.
For each article, we examine the methods to find their model assumptions, and we create a table with several fields of numerical and categorical variables. Numerical variables include the vaccine efficacy against death used in the models.
From publicly available sources
Five of the twelve models did not consider the observed waning of immunity in vaccinated populations, assuming a constant vaccine effectiveness. Some models did putatively account for waning immunity (
* Based on Table 3, scenario S3, considering the impact of maintaining public health measures but not performing vaccination for a population of 38.0 million Canadians (2021, Statistics Canada ** Based on his postulated claim that one death was averted for every 127 primary series vaccinations given. Between 3/19/2021 and 5/22/2022, 53.9% of the total US population was vaccinated
Reference
Region
Time frame
Recorded C19 deaths
Averted mortality estimates
Model vaccine efficacy death
Is vaccine efficacy assumed constant?
Code accessible?
Europe (age 60 years and older)
Up to Week 45, 2021
442,116
469,186 [129,851-733,744]
95%
Yes
Yes
Global
Week 50, 2020 to Week 49, 2021
5 469 000
14.4 million [13.7-15.9]
Adenovirus, 92%; mRNA, 95%; subunit, 96%; whole virus, 79%
Yes, accounts for decreased VE against variants
Yes
USA
Week 50, 2020 to Week 26, 2021
300, 081(25)
240,797 [200,665-281,230]
92%
Yes, accounts for decreased VE against variants
Yes
USA
Up to Week 48, 2021
800, 000
1,087,191 [950,101 - 1,231,195]
Not stated “drawn from published estimates”
Stated to account for waning immunity
No, parameters not available
USA
Up to Week 18, 2021
585,285
139,393
N/A
N/A
Model assumptions available
Canada
Up to Week 16, 2022
38,783
321,077 [175,157-764,917] *
96%
Accounts for waning VE against infections and hospitalization, VEdeath is constant
No, parameters available
USA
Week 51, 2020 to Week 22, 2021
250,000
123,200 [-74,300 -403,000]
90% VEinfection Does not use a separate VEdeath
No
Yes
Northeastern and southern USA (hypothetical increase in daily vaccine doses by 50%)
Week 40, 2020 to Week 35, 2021
324,649
158,665 [144,640-172,690]
For severe disease: Original Strain, 92%; Alpha, 94%; Beta, 97.4%; Delta, 80%
Yes, after two weeks
Yes
Israel
Week 51, 2020 to Week 14, 2021
2,859
5,532 [3,085-7,982]
Calculated using rate differences, does not require VE. But VE at 96.7%
Is not a modelled study
Code for data analysis (not simulation) is not available.
USA
Week 11, 2021 to Week 20, 2022
351,777
1.4 million **
Regression analysis, does not use VE
Accounts
Code for data analysis (not simulation) is not available.
Finland
Week 52, 2020 to Week 13, 2022
1,753
7321[6602-8084]
Regression analysis, calculates VE at 98%.
Not dependent on modeling
N/A
New York City, NY, USA
Week 51, 2020 to Week 28, 2021
9,104
8,508[7,374-9,543]
For severe disease: Original Strain, 92%; Alpha, 94%; Beta, 97.4%; Delta, 97.4%
Yes, after two weeks
Yes
In most studies, we observe the following trend. A rapid waning of protection against infection
One overlooked aspect of these counterfactual scenarios is that none of those mentioned in
The UK Office of National Statistics (ONS), estimated the NNV for prevention of severe hospitalization to be 2,500 for those 70 and older and 18,700 for those 50-59
One consideration for averted mortality modeling is that it provides a number of deaths. However, these are not differentiated by expected life years. Disability adjusted life years (DALYs) are a population health metric which accounts for the expected number of life years saved
Averting the death of an elderly person carries considerable benefit, though they will likely at most live a few more years. The death of a young person is a loss of multiple decades of healthy life. These should not be weighed the same as their impact is considerably different.
Very little justification existed for vaccinating the non-elderly without serious co-morbidities, as these populations had very low risk from COVID
Of the models explored, most had systematic biases towards overstating the effectiveness of Covid-19 vaccines in averting mortality. These estimates vary widely and are not directly comparable, but we do have access to the time periods of the estimates and the reported number of Covid-19 deaths during that time periods. As the difference between the no-vaccine situation with the vaccine situation can only manifest after vaccination begins, and the model assumptions create more divergent outcomes with the greater passage of time, the model predictions cannot be directly compared. However, most of the averted mortality estimates are on the order of the number of Covid-19 related fatalities in the region and time frame of the estimation. One major exception is the Canadian study
Given that vaccination has downsides, and the imposition of emergency measures and mandates comes with severe downsides, it is important to know the actual benefit, if any, that vaccination brings. Thus far, the models that exist are set to systematically overstate the level of averted mortality, while downplaying or denying any costs and negative risks. In the context of making informed policy decisions, it is unacceptable to emphasize the benefits of a particular intervention while downplaying or ignoring the risks. Here, modelers show systemic bias towards showing the benefits of vaccination while downplaying the risks. This analysis shows that the models overstate the averted mortality through several distinct mechanisms, by using inflated case fatality rates which overstate the danger of Covid-19, by overstating the effectiveness of vaccines against death, as well as the transmission of the virus, and by ignoring waning vaccine immunity and vaccine adverse events. Another unappreciated factor is that the age structure of the averted mortality is concentrated in the elderly, who are at most risk for Covid-19, and averted mortality values correspond to at most a few extra years of life. While this may seem like ghoulish math to some, it has been widely accepted that elderly people have already lived a full life, and saving the lives of younger people is of higher priority.
In the case of Covid-19 policy, the case was made that the young should get vaccinated, despite being at almost negligible risk. When the cost benefit analysis is not in favor of vaccinating the young for their protection, the argument shifted to one of the social duties to create herd immunity. This argument first had the issue of feasibility and fails even if you accept the alleged utilitarian argument, as the vaccines do little to stop transmission, and regular booster vaccination is associated with higher rates of infection. Secondly, while any supposed benefits accrue to the elderly, young people suffer the harm, violating bedrock bioethical principles. One cannot be coerced to undergo a medical procedure for the (supposed) benefit of another; while it may seem an extreme comparison, it is only a difference of magnitude that separates this practice from forced organ harvesting, which too purports to deliver a benefit to another at comparably minimal cost from one person. This practice is at odds with protection of human rights, and violations which are not immediately halted and punished countenance the destruction of bedrock principles of individual rights.
Additional issues that don’t quite fit into methodological categories include the inaccessibility of several of the models. While many authors and models do provide their codes and parameters for replication, some of the models providing the most sway in terms of their impact on policy makers are black box models. The Commonwealth fund model
This article provides a critical assessment of the models used to estimate the number of deaths averted by COVID-19 vaccination campaigns. The authors systematically reviewed twelve studies published between December 2020 and April 2023, spanning various countries and regions. Their analysis reveals that most models rely on assumptions that tend to overstate the benefits of vaccination. Key issues include the frequent use of constant, high vaccine efficacy values without adequately accounting for waning immunity, which is well-documented in real-world data. Only half of the reviewed models adjusted for waning immunity, and even these adjustments often favored higher estimates of averted mortality. Notably, none of the models considered potential adverse events from vaccination, such as rare but serious side effects, leading to a one-sided portrayal of vaccine benefits. Additionally, the models generally report only the number of deaths averted, without considering the age distribution or the actual life years saved, even though most averted deaths occurred among the elderly, for whom the gain in life years would be modest. Transparency is also a concern, as some influential models lack publicly available code or detailed parameters, making independent verification difficult. The magnitude of averted deaths estimated by most models is like the actual recorded COVID-19 deaths in the same period and region, though there are outliers, such as a Canadian study estimating over eight times as many averted deaths as actual deaths.
For assessing health interventions, a more common metric to use besides ‘averted deaths’ is Years of Life Saved (YLS), or QALYs or DALYs. These metrics consider the true mortality benefit of an intervention. Averted mortality can be misleading when used to measure the effectiveness of interventions. Covid-19 predominantly affects the elderly, so any realized mortality benefit will be concentrated in the elderly, resulting in low YLS. For a public safety intervention aimed at minimizing traffic fatalities for example, the ratio of YLS to averted mortality would be expected to be much higher, as the age distribution of those lives saved leans much younger. For example, only 10.3% of traffic deaths are in those aged 75 and older
In conclusion, the article underscores the need for improved epidemiological modeling in assessing the impact of COVID-19 vaccination. The current modeling landscape is characterized by assumptions that inflate the estimated number of deaths averted, insufficient attention to waning immunity, and a lack of consideration for vaccine-related adverse events and the age structure of those whose deaths were prevented. These limitations can distort public health policy, especially when such models are used to justify mandates or emergency measures. The authors call for greater rigor, transparency, and balance in future modeling efforts, advocating for realistic assumptions about vaccine efficacy over time, inclusion of potential adverse events, and consideration of the age and health status of those whose deaths are averted. Robust and replicable analyses that openly acknowledge uncertainties and limitations are essential for informed policymaking that genuinely balances the benefits and risks of mass vaccination campaigns.