what is a test negative case control study

• Research article
• Open access
• Published: 08 December 2017

A comparison of the test-negative and the traditional case-control study designs for estimation of influenza vaccine effectiveness under nonrandom vaccination

• Meng Shi 1 ,
• Qian An 1 ,
• Kylie E. C. Ainslie 1 ,
• Michael Haber 1 &
• Walter A. Orenstein 2  

BMC Infectious Diseases volume  17 , Article number:  757 ( 2017 ) Cite this article

11k Accesses

30 Citations

9 Altmetric

Metrics details

As annual influenza vaccination is recommended for all U.S. persons aged 6 months or older, it is unethical to conduct randomized clinical trials to estimate influenza vaccine effectiveness (VE). Observational studies are being increasingly used to estimate VE. We developed a probability model for comparing the bias and the precision of VE estimates from two case-control designs: the traditional case-control (TCC) design and the test-negative (TN) design. In both study designs, acute respiratory illness (ARI) patients seeking medical care testing positive for influenza infection are considered cases. In the TN design, ARI patients seeking medical care who test negative serve as controls, while in the TCC design, controls are randomly selected individuals from the community who did not contract an ARI.

Our model assigns each study participant a covariate corresponding to the person’s health status. The probabilities of vaccination and of contracting influenza and non-influenza ARI depend on health status. Hence, our model allows non-random vaccination and confounding. In addition, the probability of seeking care for ARI may depend on vaccination and health status. We consider two outcomes of interest: symptomatic influenza (SI) and medically-attended influenza (MAI).

If vaccination does not affect the probability of non-influenza ARI, then VE estimates from TN studies usually have smaller bias than estimates from TCC studies. We also found that if vaccinated influenza ARI patients are less likely to seek medical care than unvaccinated patients because the vaccine reduces symptoms’ severity, then estimates of VE from both types of studies may be severely biased when the outcome of interest is SI. The bias is not present when the outcome of interest is MAI.

Conclusions

The TN design produces valid estimates of VE if (a) vaccination does not affect the probabilities of non-influenza ARI and of seeking care against influenza ARI, and (b) the confounding effects resulting from non-random vaccination are similar for influenza and non-influenza ARI. Since the bias of VE estimates depends on the outcome against which the vaccine is supposed to protect, it is important to specify the outcome of interest when evaluating the bias.

Peer Review reports

Influenza vaccine effectiveness (VE) has to be re-estimated in every season because predominant influenza virus types, subtypes and phenotypes change from one season to the next, necessitating a new vaccine targeting different strains in most seasons. As annual influenza vaccination is now widely recommended, randomized clinical trials for estimating VE are no longer ethical in many populations, and observational studies based on patients seeking medical care for acute respiratory illnesses (ARI) are the most efficient, and hence most widely used option. However, observational studies for estimating VE are prone to multiple sources of bias.

In this paper we present a new probability model for comparing the bias and precision of VE estimates from two popular case-control study designs under nonrandom vaccination, i.e., vaccination probabilities may depend on a covariate. In both study designs, ARI patients seeking medical care who test positive for influenza infection are considered cases. In the test-negative (TN) design, ARI patients seeking medical care who test negative for influenza infection serve as controls, while in the traditional case-control (TCC) design, controls are randomly selected individuals who did not contract an ARI, usually from the same community from which the cases came. The TN design was introduced in 2007 [ 1 ], and most of the influenza VE case-control studies conducted since then have used this study design. However, TCC studies are still being used occasionally [ 2 – 4 ]. TCC studies are usually costlier and more resource intensive due to the need to recruit controls through a separate mechanism.

Estimates of VE from case-control studies may be subject to the following sources of bias:

(a) Probabilities of non-influenza ARI may depend on vaccination status: In TN studies, individuals with non-influenza ARI serve as controls. Therefore, TN studies may produce biased estimates of VE unless vaccinees and non-vaccinees are equally likely to develop non-influenza ARI. The validity of this assumption has not yet been confirmed. De Serres et al. [ 5 ] used data from randomized clinical trials to argue that this assumption is usually satisfied. However, a randomized influenza vaccine trial [ 6 ] found that vaccinees had a significantly increased risk of virologically-confirmed non-influenza infection (that may lead to ARI) as compared to those who received the placebo.

(b) Probabilities of influenza and non-influenza ARIs may depend on confounders: Covariates such as health status, age, exposure, education and socio-economic status may be associated with both the likelihood of being vaccinated and the likelihood of developing influenza and non-influenza ARIs.

(c) Vaccination may affect probability of seeking medical care in influenza patients: Several studies [ 7 – 9 ] suggest that vaccinated individuals who contract influenza may have milder symptoms than unvaccinated influenza patients, and therefore may be less likely to seek medical care. This effect of vaccination is not expected to change health-care-seeking behavior of non-influenza ARI patients.

(d) Probabilities of seeking medical care against ARIs may depend on confounders: Since only ARI patients who seek medical care may be included in a TNC study, and only influenza patients who seek care may be included as cases in TCC studies, covariates that are associated with both the likelihood of being vaccinated and the likelihood of seeking care against ARI may contribute to the bias of influenza VE estimates.

(e) Misclassification bias: Diagnostic tests for influenza viruses are not 100% sensitive and specific. Vaccination status may also be misclassified.

In this work we consider the first four sources of bias. To focus on these sources, we ignore misclassification biases which are known to result in negative bias (i.e., bias toward lower estimation of VE) and are common to all studies that rely on results of diagnostic tests.

The goal of this article is to evaluate and compare the bias and precision of estimates of VE resulting from TN and TCC studies. As we will see, the bias of VE estimates may depend on the outcome of interest, i.e., the outcome against which the vaccine is expected to protect. We consider two outcomes of interest, symptomatic influenza (SI) and medically-attended influenza (MAI). In both the TN and TCC study designs, only influenza patients seeking medical care are considered cases. Therefore, one expects these study designs to produce estimates of VE against MAI. However, the media usually reports VE estimates from these case-control studies as ‘vaccine effectiveness against influenza’, without including the ‘medically-attended’ clause. As a result, the public may interpret these estimates as the effectiveness of the vaccine against any influenza illness , i.e., VE against SI. One of the objectives of this work is to highlight the importance of (a) clearly specifying the outcome against which the vaccine is supposed to protect, and (b) understanding that the bias of a VE estimate may be different for the two outcomes of interest.

We will (a) evaluate the bias of each of the VE estimates for each of the outcomes by comparing the expected value of the estimate with the true VE, and (b) evaluate the standard errors of the VE estimates. To perform these evaluations and comparisons, we developed a detailed stepwise probability model of the process involved in collecting data in these studies and deriving VE estimates. The model includes a covariate, health status, that may be associated with both the likelihood of being vaccinated and the propensity of seeking medical care against ARI. This allows us to assess the effects of nonrandom vaccination on the bias of VE estimates.

We first describe the real-life process involved in conducting the two types of case-control studies and obtaining the estimates of VE. We then describe the model we developed to mimic this process.

The study population

The source population for both types of case-control studies consists of all individuals receiving most of their medical care at a single clinic or at a specific network of clinics. Since influenza VE varies by age, we can assume that the model pertains to a subpopulation corresponding to a single age group.

The study designs

When a member of the study population develops an ARI, s/he may decide to report to a clinic for treatment. At the clinic, the health care provider may ask the person to be tested for influenza viruses. If the person agrees then a swab is taken and sent to a laboratory for testing. In both study designs, a person who tests positive is eligible to be considered a case. In a TN study, an individual who tests negative is eligible to be considered a control. In a TCC study, controls are randomly selected members of the study population who have not developed ARI prior to their inclusion in the study. Usually, one or more controls are selected right after a case is identified. In both study designs, the vaccination status of every case or control is determined from manual or electronic records, or from oral histories.

Outcome of interest and true VE

In this work we evaluate estimates of VE when the outcome of interest is either SI or MAI. SI is sometimes called ‘influenza illness’ or ‘influenza ARI’. Surveillance for SI is needed in the entire study population, and for persons ill with compatible illnesses, samples of influenza are taken for verification. A person is considered a true case of SI if s/he has an ARI and is infected by an influenza virus. For MAI, a true case is defined as a person who is influenza-infected, develops an ARI, and seeks medical care. In both cases, the true VE is defined as one minus the ratio of the probability of the outcome of interest in vaccinees and non-vaccinees.

Estimation of VE and bias of VE estimates

In this work we focus on identifying the main sources of bias and their effects on the performance of the VE estimates. Some of these biases can be adjusted for in the analysis, but this is beyond the scope of the current work. In case-control studies, VE is usually estimated as one minus the odds ratio (OR) of being vaccinated in cases vs. controls. The bias of the estimate is defined as the difference between the expectation of the estimated VE and the true VE.

The model we developed for comparing the estimates from the two study designs follows the scheme described above with a few simplifications. We assume that (a) when a person seeks medical care for ARI then her/his probability of being tested for influenza viruses does not depend on vaccination status, health status, or on the actual cause of ARI (influenza/non-influenza); (b) given a person’s symptoms and influenza infection status, the sensitivity and specificity of the test do not depend on the tested person’s vaccination or health status; (c) a person’s vaccination status is determined without error; and (d) controls in a TCC study are selected at random from all asymptomatic individuals in the study population (See “ The study population ” section).

Our model includes a covariate, health status, and we assume that a person’s probabilities of being vaccinated, developing an ARI, and seeking medical care against ARI may be associated with her/his health status. In this way, the model generates possible confounding effects linking vaccination status with the probabilities of being included in the study and of becoming a case or a control.

The model consists of five steps, where the value of a single variable is determined at each step. The probability distribution of this variable may depend on the values of the variables from the previous steps. Below we define the five steps, the associated variables, and the probabilities determining each variable’s distribution.

Step 1: Health status

A person can be classified as “healthy” or “frail”. Define a binary variable X, where X =1 for a “healthy” person and X =0 for a “frail” person. Denote π = P ( X =1).

Step 2: Vaccination

A person may be vaccinated against influenza. Define a binary variable V , where V =1 for a vaccinated person. The probability of being vaccinated may depend on health status; therefore, denote α x = P ( V =1| X = x ), x =0,1.

Step 3: Influenza infection and ARI

During the influenza season, a person may become infected with an influenza virus and develop an ARI. This outcome is referred to as “influenza ARI” (FARI), where “F” stands for flu. A person may also develop an ARI not resulting from influenza infection. This outcome is referred to as “non-influenza ARI” (NFARI). We therefore define an outcome variable Y with 3 categories as follows: Y =0 for no ARI, Y =1 for NFARI, and Y =2 for FARI. The distribution of Y depends on the person’s vaccination status, V , and health status, X . We denote β vx = P ( Y =1| V = v , X = x ), v =0,1, x =0,1 and γ vx = P ( Y =2| V = v , X = x ) for v =0,1, x =0,1 with β vx + γ vx ≤1 for all v , x . Here we assume the “leaky vaccine” model, in which the vaccine provides a reduction in the probability of influenza transmission to the vaccinated person, rather than complete immunity [ 10 ]. Under this model, a vaccinee has a lower probability of becoming infected than a non-vaccinee, but is not rendered completely immune from influenza infection.

Step 4: Seeking medical care for ARI

A person with ARI may seek medical care and, in this case, be tested for influenza viruses. We define a binary variable M with M =1 for a person seeking medical care for her/his ARI. The probability of this event depends on Y (only individuals with ARI seek medical care), and it may be different for FARI and NFARI patients. In addition, the conditional distribution of M given Y may depend on X and V . We therefore define δ yvx = P ( M =1| Y = y , V = v , X = x ), where y =1,2, v =0,1 and x =0,1.

In order to reduce the number of parameters, we make two simplifying assumptions regarding the probabilities of seeking medical care: (1) the effect of health status on probability of seeking medical care does not depend on vaccination status or type of ARI; (2) the effect of vaccination status on probability of seeking medical care does not depend on health status (but it may depend on type of ARI).

Define a “standard person” as a person with X = 0 and V = 0. For a “standard person”, we define δ SN , δ SF as follows:

δ SN = P ( M =1| Y =1, V =0, X =0)= δ 100

δ SF = P ( M =1| Y =2, V =0, X =0)= δ 200

In addition, we define two multipliers:

λ = multiplier for x = 1; λ does not depend on V and Y.

Ψ F = multiplier for v=1 only when y=2; Ψ F does not depend on X.

λ is the ratio of the probabilities of seeking medical care comparing a healthy and a frail person. Ψ F is the ratio of the probabilities of seeking care comparing a vaccinated and unvaccinated influenza ARI patient.

Then, { δ yvx } can be written in terms of δ SN , δ SF and the multipliers λ , Ψ F as follows:

δ 100 = δ SN , δ 101 = δ SN ∗ λ , δ 110 = δ SN , δ 111 = δ SN ∗ λ .

δ 200 = δ SF , δ 201 = δ SF ∗ λ , δ 210 = δ SF ∗ Ψ F , δ 211 = δ SF ∗ λ ∗ Ψ F .

Note: The multiplier Ψ F reflects the effect of severity of ARI in an influenza infected person. We assume that vaccination may reduce severity of symptoms, hence a vaccinated influenza patient may be less likely to seek care than an unvaccinated patient.

Step 5: Testing for influenza infection.

Although only individuals who seek medical care for an ARI are tested for influenza infection, it will be convenient to define a binary variable T as the (possibly unobserved) test result for any person with an ARI, regardless of whether or not s/he is actually tested. Define T =1 ( T =0) if a person would test positive (negative) for influenza if tested. Because of assumption (b) above, the probability of testing positive given the person’s influenza infection status does not depend on X , V , or M . Denote τ y = P ( T =1| Y = y ) for y =1,2. Note that τ 1 is one minus the test’s specificity and τ 2 is the test’s sensitivity. In this study, we assume the test has 100% sensitivity and 100% specificity, i.e. P ( T =1| Y =1)= τ 1 =0 and P ( T =1| Y =2)= τ 2 =1.

Figure  1 shows the directed acyclic graph (DAG) of the model. Recent papers by Sullivan et al. [ 11 ] and Lipsitch et al. [ 12 ] discuss the use of DAGs to explore sources of bias of VE estimates from TN studies. A summary of the variables and parameters in our model is given in Table  1 .

DAG of the model

True VE in our model

When we evaluate the true VE, we assume that vaccination is done at random , i.e. for true VE we assume that vaccination status does not depend on health status X ( α 0 = α 1 = α ).

The true VE against SI is:

The true VE against MAI is:

Using the parameters defined above, V E T SI and V E T MAI can be written as:

The proofs of these results can be found in Appendix 1 .

Estimates of VE in our model

In both the TN and TCC study designs, VE is estimated as one minus the odds ratio (OR) in the C × V table cross-classifying the individuals included in the study, where C is a binary indicator of case/control status with C =1 for a case. For convenience, the TN and TCC studies will be represented by the letters A and B, respectively. In a TN study, the case/control variable is denoted C A , where ( C A =1)=( M =1, T =1) and ( C A =0)=( M =1, T =0). Then the estimate of VE is: V E A =1− O R A , where

Note that all the probabilities condition on M =1 as only individuals who seek medical care for ARI are included in the TN study.

In a TCC study, the case/control variable is denoted C B . Cases are defined in the same way as in the TN study, i.e., ( C B =1)=( M =1, T =1)=( C A =1). Controls are individuals included in a random sample drawn from all the asymptomatic individuals in the study population. In other words, ( C B =0) is a random subset of ( Y =0). In addition, we define a binary variable B indicating whether or not a person is included in the TCC study, i.e., ( B =1)=( C B =1or C B =0). The VE estimate is based on the OR in the C B × V table when all the probabilities condition on B =1: V E B =1− O R B , where

Note that in a real-life study, the odds ratios are estimated from the relative frequencies of the corresponding events, rather than from their (unknown) probabilities. Therefore, the model-based estimates of VE defined above are actually the expected values of the observed estimates. For convenience we will continue to refer to them as “the VE estimates”.

Using the parameters defined above, V E A and V E B can be written as follows:

The proofs can be found in Appendix 2 .

Bias and standard errors of estimates

The bias of an estimate of VE is the difference between the expected value of the estimate and the true VE. As the true VE depends on the outcome of interest (SI or MAI), the bias of each estimated VE will be evaluated separately for each of the two outcomes.

In Appendix 3 we use approximations based on the “Delta method” for the standard errors (SEs) of odds ratios [ 13 ] to derive expressions for the SEs of both VE estimates in terms of the parameters and the corresponding sample size(s). For evaluating the SEs we consider the observed odds ratios, where the probabilities are replaced by the corresponding observed relative frequencies.

The values of bias reported in the text and tables represent absolute numbers. For example, if the true VE is 60% (i.e., 0.6) and the range of bias (-0.40, -0.20). This means that the estimated VE varies from 0.20 (underestimating the true VE = 0.6 by 0.40) to 0.80 (overestimating the true VE by 0.20).

Probability ratios

Next, we define a few probability ratios comparing vaccinees and non-vaccinees or healthy and frail individuals. These ratios will be helpful in the presentation of the results (see Table  1 for a full list of the notations used in this paper).

\(\rho _{\beta } = {\frac {\beta _{1x}}{\beta _{0x}}}\) , the ratio of the probabilities of NFARI comparing a vaccinated and an unvaccinated person of the same health status.

\(\eta _{\beta } = {\frac {\beta _{v1}}{\beta _{v0}}}\) , the ratio of the probabilities of NFARI comparing a healthy and a frail person of the same vaccination status.

\(\rho _{\gamma } = {\frac {\gamma _{1x}}{\gamma _{0x}}}\) , the ratio of the probabilities of FARI comparing a vaccinated and an unvaccinated person of the same health status.

\(\eta _{\gamma } = {\frac {\gamma _{v1}}{\gamma _{v0}}}\) , the ratio of the probabilities of FARI comparing a healthy and a frail person of the same vaccination status.

The parameters λ and Ψ F defined earlier are also probability ratios:

\(\lambda = {\frac {\delta _{yv1}}{\delta _{yv0}}}\) The ratio of the probabilities of seeking medical care comparing a healthy and a frail person of the same vaccination status. We assume that this ratio is the same for FARI and NFARI patients.

\(\Psi _{F} = {\frac {\delta _{21x}}{\delta _{20x}}}\) The ratio of the probabilities of seeking medical care comparing a vaccinated and an unvaccinated FARI patient of the same health status.

Table  2 presents the main sources of bias that can be identified from our model. The absence of bias A is essential for the validity of the TN design, since the VE estimate from this design is based on comparing the odds of being vaccinated in FARI patients (cases) and NFARI patients (controls). This bias may be a result of virus interference [ 6 ] (if vaccinees are more likely than non-vaccinees to contract NFARI, then the estimated VE will be falsely high). Biases B1 and B2 represent the effects of health status on the probabilities of NFARI and FARI, respectively. These effects, which are sometimes called the ‘ healthy vaccinee effect ’, represent the confounding resulting from association of health status with the probability of exposure (vaccination) and the outcome. Bias BS is a special case of B 1∩ B 2. It results when health status affects both the probabilities of FARI and NFARI but the risk ratios comparing a healthy and a frail person are the same for the both types of ARIs. Bias C represents the effect of vaccination status on the probability of seeking care in patients with SI. This effect may be due to less severe symptoms in vaccinated persons compared to unvaccinated ones. As stated earlier, we assume perfect sensitivity and specificity of the influenza test ( τ 1 =0, τ 2 =1), as it is well-known that misclassifications result in negatively-biased estimates of effectiveness.

Sources of bias

We first state conditions for the unbiasedness of the VE estimate based on the TN design. The proofs of these results can be found in Appendix 4 .

Under random vaccination ( α 0 = α 1 ), the estimate of VE when the outcome of interest is SI is unbiased if biases A and C are absent. When the outcome of interest is MAI, the estimate of VE is unbiased if bias A is absent.

Under non-random vaccination ( α 0 ≠ α 1 ), the estimate of VE when the outcome of interest is SI is unbiased if biases A, B1, B2, and C are absent. When the outcome of interest is MAI, the estimate of VE is unbiased if biases A, B1, and B2 are absent.

It is interesting to note that the absence of any source of bias, the OR-based VE estimate from a TN study is unbiased even if the ‘rare disease’ assumption is not satisfied, while the OR-based estimate from a TCC study is biased. To show this, let’s use the following simplified notation: α = probability of being vaccinated, β = probability of NFARI, γ 0 and γ 1 = probabilities of FARI in unvaccinated and vaccinated, respectively, and δ = probability of seeking care. Then the true VE is 1− ρ , where ρ = γ 1 / γ 0 is the risk ratio. In a TN study, the probabilities of vaccinated and unvaccinated cases are α ∗ δ ∗ γ 1 , and (1− α ) ∗ δ ∗ γ 0 , respectively. The corresponding probabilities of controls are α ∗ δ ∗ β , and (1− α ) ∗ δ ∗ β , respectively. Then the OR in the table of case-control status by vaccination status equals to ρ , i.e. the true risk ratio, implying that the estimated VE is unbiased. In a TCC studies, the probabilities of cases are the same as in the TN study, while the probabilities of vaccinated and unvaccinated controls (individuals without ARI) are ϕ ∗ α ∗ (1− γ 1 − β ) and ϕ ∗ (1− α ) ∗ (1− γ 0 − β ), respectively, where ϕ is the sampling fraction of controls. Hence, the OR in the TCC study is [ ρ ∗ (1− γ 0 − β )]/(1− ρ ∗ γ 0 − β ). This OR is less than ρ (the true RR) if ρ > 0, hence the estimated VE exceeds the true VE in a TCC study as long as the true VE is positive.

Next we explore the magnitude of the effects of various sources of bias and their combinations. We consider three scenarios for vaccination probabilities (see Table  3 ). In Table  4 we present the range and the maximum absolute value of the bias of VE estimates resulting from TN and TCC studies under the three vaccination scenarios and various combinations of sources of bias. For these results we used the following baseline values of some of the parameters: π =0.7, β 00 =0.2, γ 00 =0.1, δ SN =0.2, δ SF =0.3, ρ γ =0.4. π is the probability of being ‘healthy’; β 00 and γ 00 are the probabilities of NFARI and FARI, respectively, for an unvaccinated ‘frail’ person; δ SN and δ FN are the probabilities of seeking medical care for NFARI and FARI, respectively, for an unvaccinated ‘frail’ person; ρ γ is the risk ratio comparing the probability of FARI for a vaccinated and an unvaccinated person - thus, the true VE against SI is 1 - 0.4 = 0.6 (60%). The values of β ’s, γ ’s are based on data from various randomized placobo-controlled trials (see [ 14 ], Table A1), and the values of δ are based on data from several observational studies. In all the tables, figures and examples, values of VE are presented as fractions, rather than percentages.

In the calculations for Tables  4 and 5 , when a source of bias was present we used a reasonable range for the corresponding probability ratio. When bias A was present, ρ β was allowed to vary from 0.5 to 2.0. For biases B1, B2, and BS, we allowed η β and η γ to vary between 0.5 and 1.0, since one would not expect frail persons to have lower probabilities of ARI, compared to healthy persons. For bias C, the ratio Ψ F could vary between 0.5 to 1.0, since one would expect vaccination to reduce the probability that a person with SI will seek medical care compared to a person with ARI resulting from a different pathogen. For bias D, we let λ vary between 0.5 to 2.0.

For each combination of two or more sources of bias, we calculated the minimum, mean, and maximum of the bias and the absolute values of the bias by allowing the probability ratios that are not fixed to vary independently in the ranges specified above. For example, when biases A, B1, and B2 are absent, we used ρ β = η β = η γ =1,0.5≤ Ψ F ≤1,0.5≤ λ ≤2.

Summary of results

The impact of sources of bias.

Our model allows us to evaluate the impact of the sources of bias listed in Table  2 . Each source of bias is a result of a possible effect of vaccination or health status on the probability of FARI or NFARI or seeking care. Below we summarize our results for each of the sources of bias. We also use numerical examples to illustrate the magnitude and direction associated with each source of bias. Unless otherwise specified, the true VEs against SI and MAI are 0.6 (60%). In each of these examples we assume that only one source of bias is present.

Vaccination affects the probability of NFARI (bias A)

This bias does not depend on vaccination scenario nor on the outcome of interest (SI or MAI).

Estimates of VE from TN studies may suffer from severe bias.

This effect also affects the bias of VE estimates from TCC studies, though to a lesser extent.

Example: As the ratio of the probability of NFARI comparing vaccinated and unvaccinated persons varies from 0.5 to 2.0, VE estimates from TN studies range from 0.2 to 0.8, respectively, while VE estimates from TCC studies range from 0.67 to 0.50, respectively (Fig.  2 ).

True and estimated VEs as a function of R1 = P(NFARI if vaccinated)/P(NFARI if unvaccinated) when only bias A is present

Health status affects the probabilities of FARI and NFARI (biases B1, B2 - the ‘ healthy vaccinee effect ’)

The bias does not depend on the outcome of interest (SI or MAI).

Under non-random vaccination, these effects may result in substantial bias of VE estimates from TN or TCC studies. However, this bias is usually less severe compared to the biases resulting from sources A, C and D.

If the effect of health status on the probability of ARI is the same for FARI and NFARI, i.e., bias BS is present, then the TN-based estimates of VE are unbiased.

Example: Suppose that the probabilities of vaccination are 0.8 and 0.4 for healthy and frail persons, respectively. Consider three cases regarding the risk ratios P(ARI in a healthy person) / P(ARI in a frail person): (a) When these risk ratios are 0.5 for NFARI and 0.8 for FARI, then the estimated VEs from TN and TCC studies are 0.51 and 0.67, respectively. (b) When the risk ratios are 0.8 for NFARI and 0.5 for FARI, then estimated VEs from TN and TCC studies are 0.67 and 0.72, respectively. (c) When the risk ratios for NFARI and FARI are equal and their common value ranges from 0.5 to 1.0, then estimated VEs from TN studies are always unbiased (i.e., they equal 0.6), while estimates from TCC studies range from 0.63 to 0.73. In Figs.  3 and 4 , we set the risk ratio for NFARI to 0.75 and let the risk ratio for FARI vary between 0.5 to 1.0.

True and estimated VE’s when only biases B1 and B2 are present. We set the risk ratio P(NFARI if healthy)/P(NFARI if frail) =0.75 and let the risk ratio R2 = P(FARI if healthy)/P(FARI if frail) vary between 0.5 to 1.0. The probabilities of vaccination are 0.4 and 0.8 for healthy and frail persons, respectively

True and estimated VEs when only biases B1 and B2 are present. We set the risk ratio P(NFARI if healthy)/P(NFARI if frail) =0.75 and let the risk ratio R2 = P(FARI if healthy)/P(FARI if frail) vary between 0.5 to 1.0. The probabilities of vaccination are 0.8 and 0.4 for healthy and frail persons, respectively

Vaccination affects the probability of seeking medical care for FARI, but it does not affect the probability of seeking care for NFARI (bias C)

When this effect is present then the true VEs against SI and MAI may be different as the vaccine directly affects (reduces) the probability of seeking care in influenza cases, but not in controls. Thus the estimates’ bias may depend on the outcome of interest.

If all other sources of bias are absent, the bias of VE estimates does not depend on the vaccination scenario.

Estimates of VE from TN or TCC studies may be severely biased when the outcome of interest is SI.

When the outcome of interest is MAI, estimates of VE from TN studies are unbiased, while the bias of estimates from TCC studies is usually small and is not affected by the magnitude of the effect underlying this source of bias.

Example: Let the ratio R = P(seeking medical care against FARI if vaccinated) / P(seeking medical care against FARI if unvaccinated) vary from 0.5 to 1.0. Then the true VE against SI remains fixed at 0.6, while the true VE against MAI varies with R from 0.8 to 0.6. The estimated VEs from TN studies equal the true VE against MAI for all values of R, while the estimated VEs from TCC studies vary from 0.82 to 0.63 (see Fig.  5 ). For example, when R =0.5 then the true VE against MAI is 0.80, and the VE estimates from TN and TCC studies are 0.80 and 0.82, respectively. This translates into severe bias when the outcome of interest is SI but small bias when the outcome of interest is MAI.

True and estimated VEs when only bias C is present as function of R3 = P(seeking medical care against FARI if vaccinated)/P(seeking medical care against FARI if unvaccinated)

Health status affects the probabilities of seeking care against FARI and NFARI (bias D)

The bias of VE estimates does not depend on the outcome of interest (SI or MAI).

In the absence of other sources of bias, VE estimates from TN studies are unbiased regardless of the vaccination scenario.

Under non-random vaccination, this effect may result in substantial bias in VE estimates from TCC studies.

Example: We assume that the probabilities of seeking care do not depend on vaccination status. As the ratio of the probabilities of seeking care comparing healthy and frail individuals varies from 0.5 to 2.0, VE estimates from TN studies remain fixed at 0.6 (i.e., they are unbiased) under both random and non-random vaccination. When the probabilities of vaccination are 0.8 and 0.4 for healthy and frail persons, respectively, the VE estimates from TCC studies vary from 0.72 to 0.53 (Figs.  6 and 7 ).

True and estimated VEs when only bias D is present as function of R4 = P(seeking medical care if healthy)/P(seeking medical care if frail). Probabilities of vaccination are 0.8 and 0.4 for healthy and frail persons, respectively

True and estimated VEs when only bias D is present as function of R4 = P(seeking medical care if healthy)/P(seeking medical care if frail). Probabilities of vaccination are 0.4 and 0.8 for healthy and frail persons, respectively

In addition, we found that in some cases the true VEs against SI and MAI are different. Hence, the bias of VE estimates may depend on the outcome against which the vaccine is supposed to protect. For example, if the only sources of bias are BS, C, and D then, the VE estimate from TN studies is unbiased when considering effectiveness against MAI. The same estimate may overestimate the true VE against SI by 0.20 (i.e. 20%).

Comparison of the bias of VE estimates from TN and TCC studies:

If one is concerned that vaccination may affect the probability of non-influenza ARI, then one should prefer the TCC study design. However, TCC-based VE estimates may still be biased in this case. For example, when the ratio of the probability of NFARI comparing a vaccinated and an unvaccinated person is 0.5, then the bias of VE estimate from TN study is -0.4 while the bias of VE estimate from TCC study is 0.07.

Under non-random vaccination, effects of health status on probabilities of influenza and non-influenza ARI (the ‘ healthy vaccinee effect ’) may bias VE estimates from both study designs. In general, TN-based estimates perform slightly better than TCC-based estimates when this effect is believed to be the main source of bias. If the effect of health status is similar for FARI and NFARI, then the TN design produces less biased estimates compared to the TCC design. For example, suppose the probabilities of vaccination are 0.4 and 0.8 for healthy and frail persons, respectively. When the risk ratios for NFARI and FARI are both 0.75, then the VE estimate from TN study is unbiased, while the bias of VE estimate from TCC study is 0.07.

If one assumes that vaccination does not affect the probability of non-influenza ARI but one is concerned that vaccinated influenza patients are less likely to seek care than unvaccinated patients (because of reduced symptoms severity), then VE estimates may suffer from severe bias in both study designs when the outcome of interest is SI. In this case, the bias of TN-based estimates may be somewhat smaller than that of TCC-based estimates. This source of bias does not affect VE estimates when the outcome of interest is MAI! For example, suppose that the ratio comparing vaccinated and unvaccinated FARI cases w.r.t. the probability of seeking medical care is 0.5. When the outcome of interest is SI, then the bias of a VE estimate from TN study is 0.2 and the bias of a VE estimate from TCC study is 0.22. When the outcome of interest is MAI, then the VE estimate from a TN study is unbiased, while the bias of a VE estimate from a TCC study is 0.02.

Under non-random vaccination, the TN study design is preferable to the TCC design if one is concerned about bias resulting from possible effect of a person’s health status on her/his probability of seeking care against ARI. For example, suppose that the probabilities of vaccination are 0.8 and 0.4 for healthy and frail persons, respectively. When the ratio of the probabilities of seeking medical care comparing healthy and frail persons is 0.5, then the VE estimate from TN study is unbiased while the bias of VE estimate is 0.12.

Precision of VE estimates

Table  5 presents the standard errors of VE estimates from TN and TCC studies. From this table we conclude that:

Non-random vaccination may reduce precision of VE estimates.

If the probability of NFARI is associated with vaccination status, then VE estimates from TN studies are somewhat less precise compared to VE estimates from TCC studies, although the differences in precision were small.

If the probability of NFARI is not associated with vaccination status, then the precision of VE estimates from TN and TCC studies are similar.

Discussion and conclusions

We developed a new model for the evaluation of the bias and precision of influenza estimates from case-control studies. The new model is more comprehensive than previously suggested models [ 5 , 14 – 18 ] for the following reasons:

It allows assessment of the impact of non-random vaccination.

It incorporates a confounder (health status) which links vaccination status with the probabilities of ARI and of seeking medical care for these ARIs.

By including parameters corresponding to the probabilities of seeking medical care, the model allows us to examine the effect of association of these probabilities with vaccination and health status on the bias of VE estimates.

The model allows evaluating and comparing the precision of VE estimates.

Some of the sources of bias discussed here have been identified and addressed in earlier publications, but, to our best knowledge, none of the previous papers present a comprehensive discussion of all the possible sources of bias that may arise under a given model. In addition, the current model attributes the associations between the various factors involved in estimation of VE (vaccination, contracting influenza and non-influenza ARIs and seeking medical care) to an underlying covariate. Previously published models, including an earlier version of our model [ 14 ], included parameters representing these associations but these associations were not based on a common underlying factor. Therefore, we believe that the current results and conclusions may differ from those derived from less structured models.

Our calculations confirm earlier findings [ 15 ] that when the probability of non-inluenza ARI depends on vaccination status, VE estimates from test-negative studies may be severely biased. However, even when this probability is not affected by vaccination, VE estimates from the two types of case-control studies considered in this work may suffer from substantial bias. In addition to the well-known ‘ healthy vaccinee effect ’ (probabilities of vaccination and of ARI depend on health status), bias of VE estimates may result from heterogeneities in health-care-seeking behaviors. Specifically, if vaccination reduces the probability that an influenza patient seeks medical care (because her/his symptoms are less severe than those of an unvaccinated influenza patient) then VE estimates from TN or TCC studies may grossly overestimate the true VE against SI. On the other hand, when the outcome of interest is MAI then the biases resulting from vaccine-related reduction in symptoms’ severity are very small. Recent papers [ 7 – 9 ] found evidence of vaccine-associated reduction in influenza patient’s symptoms severity. The effects of health-care-seeking behaviors on VE estimates from studies in which only ARI patients who seek medical care may become cases need to be further investigated.

The results of this work lead to the following conclusions:

In general, estimates of influenza VE from case-control studies where only ARI patients seeking medical care are tested for influenza infection may suffer from severe bias, i.e. an absolute bias of 20% or more, especially when the outcome of interest is SI.

The bias of VE estimates may depend on the outcome against which the vaccine is supposed to protect. When the outcome of interest is MAI, seeking medical care is a component of the outcome. In other words, the true VE against MAI reflects the vaccine effect on seeking medical care and on contracting influenza. This explains why true VE against MAI may differ from true VE against SI. When bias C is present, the vaccine directly affects (reduces) the probability of seeking care in influenza cases, but not in controls. As a result, VE against MAI is lower than VE against SI.

Influenza VE estimates from TN studies are usually presented as ‘VE against medically-attended influenza’. However, the media and lay persons may interpret these VE estimates simply as the protective effectiveness of vaccination against contracting influenza illness, i.e. VE against SI. Health authorities and the public should be made aware of this distinction.

When the outcome of interest is SI, the TN design provides valid estimates (i.e., no or small bias) if the following assumptions are satisfied: (a) vaccination does not affect the probability of non-influenza ARI, (b) effects of confounding variables on the probabilities of influenza and non-influenza ARI are similar, and (c) vaccination does not affect the probabilities of seeking medical care for influenza ARI due to reduced severity of symptoms. When the outcome of interest is MAI, then only assumptions (a) and (b) are necessary for obtaining a valid VE estimate from a TN study.

Estimates of VE from TCC studies have small bias when the outcome of interest is SI if assumptions (a) and (c) are satisfied, assumption (b) is replaced by the stronger assumption (b*) of no presence of confounding, and the additional assumption (d) that the probabilities of seeking medical care for ARI are not affected by potential confounders is satisfied. When the outcome of interest is MAI, then TCC-based estimates of VE have small bias under assumptions (a), (b*), and (d).

It is important to collect more data on health-care-seeking behaviors of ARI patients and to study the effects of vaccination and potential confounders on these behaviors.

In summary, the test negative design produces less biased VE estimates, compared to the traditional case-control design provided that vaccination does not modify the probability of non-influenza ARI. However, this very popular study design may still produce biased estimates of influenza VE, especially when the outcome of interest is symptomatic influenza. One can expect monitored cohort studies , where every study participant reporting an ARI is tested for influenza infection, to provide less biased estimates of VE against SI. In a future publication we plan to compare the bias of these cohort studies, which are much more expensive, with that of TN studies.

Our study has a few limitations:

In order to focus on bias associated with the study designs, we ignored bias resulting from misclassification of infection and vaccination status.

Our model does not account for the dynamics of outbreaks of influenza and other ARI-causing infections.

We only consider unadjusted VE estimates as we tried to focus on sources of bias rather than on how one can reduce bias using standard or novel statistical techniques [ 19 ].

In the future we plan to improve the model by incorporating dynamics of the related processes. We also plan to use stochastic simulations to assess bias and precision of influenza VE estimates for other study designs (e.g. cohort studies) and to propose new study designs resulting in less biased VE estimates.

True VE‘s in our model

Since, for true VE, we have: α 0 = α 1 . We can get,

we can get,

Model-based estimates of VE

The model-based estimate from TN study is:

O R A can be written as:

The model-based estimates from TCC study is:

O R B can be written as:

and P ( Y =0| V = v , X = x )=1− P ( Y =1| V = v , X = x )− P ( Y =2| V = v , X = x )=1− γ vx − β vx , so we have:

Standard errors of the VE estimates

For TN study, the approximate standard error of V E A is:

where N A is the number of persons who were tested for influenza (M=1), i.e., the total sample size for the TN study. The probabilities ( \(p^{A}_{V1}\) , \(p^{A}_{01}\) , \(p^{A}_{11}\) ) can be written in terms of the parameters defined earlier.

Base on what we got earlier, we know

Therefore, we have

In the TCC study, the approximate standard error of V E B is:

where \(N^{b}_{C1}\) is the number of cases and \(N^{b}_{C0}\) is the number of controls. The probabilities \(\left (p^{B}_{10},p^{B}_{11}\right)\) can be written in terms of the parameters defined earlier:

Unbiasness under random and non-random vaccination

Unbiasness under random vaccination.

If the vaccination is done at random, then α 0 = α 1 . The VE estimates can be written as:

(1) If ρ β = Ψ F = 1, and one of the following conditions is satisfied, then V E A = V E T SI .

Since \(\rho _{\beta _{x}} = \Psi _{F}\) = 1, then \({{\beta _{10}}\over {\beta _{00}}} = {{\beta _{11}}\over {\beta _{01}}} = \Psi _{F}\) = 1. We have:

If (a) λ =1 is satisfied, so

If (b) η γ =1 is satisfied, then γ 01 = γ 00 and γ 11 = γ 10 . Thus,

(2) If ρ β =1, then V E A = V E T MAI □

Since ρ β =1, then \({{\beta _{10}}\over {\beta _{00}}} = {{\beta _{11}}\over {\beta _{01}}} = 1\) .

(3) If λ =1 and 1− γ 1 x − β 1 x = Ψ F (1− γ 0 x − β 0 x ), where x =0,1, then V E B = V E T SI □

Since 1− γ 1 x − β 1 x = Ψ F (1− γ 0 x − β 0 x ), where x =0,1, and λ =1, then

(4) If γ 1 x + β 1 x = γ 0 x + β 0 x , where x =0,1, then V E B = V E T MAI □

Since γ 1 x + β 1 x = γ 0 x + β 0 x , x =0,1, then

Unbiasness under non-random vaccination

If the vaccination is not done at random, then α 0 ≠ α 1 .

(5) If ρ β = η β = η γ = Ψ F =1, then V E A = V E T SI .

Since ρ β = η β = η γ =1, then β 00 = β 10 = β 01 = β 11 = Δ β , γ 01 = γ 00 and γ 11 = γ 10 . Thus,

(6) If ρ β =1 and η β = η γ , then V E A = V E T MAI . □

Since η β = η γ , so \({{\beta _{01}}\over {\beta _{00}}} = {{\beta _{11}}\over {\beta _{10}}} = {{\gamma _{01}}\over {\gamma _{00}}} = {{\gamma _{11}}\over {\gamma _{10}}}\) . Then we have: \({\gamma _{11}\over {\beta _{11}}} = {\gamma _{10}\over {\beta _{10}}} \stackrel {\Delta }{=} a, {\gamma _{00}\over {\beta _{00}}} = {\gamma _{01}\over {\beta _{01}}} \stackrel {\Delta }{=} b\) , and \({{\beta _{10}}\over {\beta _{00}}} = {{\beta _{11}}\over {\beta _{01}}} = 1\) . Then:

So, V E A = V E T MAI . □

Abbreviations

Acute respiratory illness

Influenza ARI

• Medically-attended influenza

Non-influenza ARI

Standard error

• Symptomatic influenza

Traditional case-control

Test-negative

Vaccine effectiveness

Skowronski DM, Masaro C, Kwindt TL, Mak A, Petric M, Li Y, Sebastian R, Chong M, Tam T, De Serres G. Estimating vaccine effectiveness against laboratory-confirmed influenza using a sentinel physician network: results from the 2005-2006 season of dual a and b vaccine mismatch in canada. Vaccine. 2007; 25(15):2842–51. doi: 10.1016/j.vaccine.2006.10.002 .

Article   CAS   PubMed   Google Scholar  

Belongia EA, Kieke BA, Donahue JG, Greenlee RT, Balish A, Foust A, Lindstrom S, Shay DK, Marshfield Influenza Study Group. Effectiveness of inactivated influenza vaccines varied substantially with antigenic match from the 2004-2005 season to the 2006-2007 season. J Infect Dis. 2009; 199(2):159–67. doi: 10.1086/595861 .

Article   PubMed   Google Scholar  

Choi WS, Noh JY, Seo YB, Baek JH, Lee J, Song JY, Park DW, Lee JS, Cheong HJ, Kim WJ. Case-control study of the effectiveness of the 2010-2011 seasonal influenza vaccine for prevention of laboratory-confirmed influenza virus infection in the korean adult population. Clin Vaccine Immunol. 2013; 20(6):877–1. doi: 10.1128/CVI.00009-13 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Havers F, Sokolow L, Shay DK, Farley MM, Monroe M, Meek J, Daily Kirley P, Bennett NM, Morin C, Aragon D, Thomas A, Schaffner W, Zansky SM, Baumbach J, Ferdinands J, Fry AM. Case-control study of vaccine effectiveness in preventing laboratory-confirmed influenza hospitalizations in older adults, united states, 2010-2011. Clin Infect Dis. 2016; 63(10):1304–11. doi: 10.1093/cid/ciw512 .

De Serres G, Skowronski DM, Wu XW, Ambrose CS. The test-negative design: validity, accuracy and precision of vaccine efficacy estimates compared to the gold standard of randomised placebo-controlled clinical trials. Euro Surveill. 2013; 18(37). https://doi.org/10.2807/1560-7917.ES2013.18.37.20585 .

Cowling BJ, Fang VJ, Nishiura H, Chan KH, Ng S, Ip DKM, Chiu SS, Leung GM, Peiris JSM. Increased risk of noninfluenza respiratory virus infections associated with receipt of inactivated influenza vaccine. Clin Infect Dis. 2012; 54(12):1778–83. doi: 10.1093/cid/cis307 .

Article   PubMed   PubMed Central   Google Scholar  

Castilla J, Godoy P, Dominguez A, Martinez-Baz I, Astray J, Martin V, Delgado-Rodriguez M, Baricot M, Soldevila N, Maria Mayoral J, Maria Quintana J, Carlos Galan J, Castro A, Gonzalez-Candelas F, Garin O, Saez M, Tamames S, Pumarola T, Influenza CCC. Influenza vaccine effectiveness in preventing outpatient, inpatient, and severe cases of laboratory-confirmed influenza. Clin Infect Dis. 2013; 57(2):167–75. doi: 10.1093/cid/cit194 .

VanWormer JJ, Sundaram ME, Meece JK, Belongia EA. A cross-sectional analysis of symptom severity in adults with influenza and other acute respiratory illness in the outpatient setting. BMC Infect Dis. 2014; 14:231. doi: 10.1186/1471-2334-14-231 .

Deiss RG, Arnold JC, Chen WJ, Echols S, Fairchok MP, Schofield C, Danaher PJ, McDonough E, Ridoré M, Mor D, Burgess TH, Millar EV. Vaccine-associated reduction in symptom severity among patients with influenza a/h3n2 disease. Vaccine. 2015; 33(51):7160–7. doi: 10.1016/j.vaccine.2015.11.004 .

Haber M, Longini Jr IM, Halloran ME. Measures of the effects of vaccination in a randomly mixing population. Int J Epidemiol. 1991; 20(1):300–10.

Sullivan SG, Tchetgen Tchetgen EJ, Cowling BJ. Theoretical basis of the test-negative study design for assessment of influenza vaccine effectiveness. Am J Epidemiol. 2016; 184(5):345–53. doi: 10.1093/aje/kww064 .

Lipsitch M, Jha A, Simonsen L. Observational studies and the difficult quest for causality: lessons from vaccine effectiveness and impact studies. Int J Epidemiol. 2016. doi: 10.1093/ije/dyw124 .

Agresti A. Categorical Data Analysis, 3rd ed. Wiley series in probability and statistics, vol. 792. Hoboken, NJ: Wiley; 2013.

Google Scholar  

Haber M, An Q, Foppa IM, Shay DK, Ferdinands JM, Orenstein WA. A probability model for evaluating the bias and precision of influenza vaccine effectiveness estimates from case-control studies. Epidemiol Infect. 2015; 143(7):1417–26. doi: 10.1017/S0950268814002179 .

Orenstein EW, De Serres G, Haber MJ, Shay DK, Bridges CB, Gargiullo P, Orenstein WA. Methodologic issues regarding the use of three observational study designs to assess influenza vaccine effectiveness. Int J Epidemiol. 2007; 36(3):623–31. doi: 10.1093/ije/dym021 .

Ferdinands JM, Shay DK. Magnitude of potential biases in a simulated case-control study of the effectiveness of influenza vaccination. Clin Infect Dis. 2012; 54(1):25–32. doi: 10.1093/cid/cir750 .

Foppa IM, Haber M, Ferdinands JM, Shay DK. The case test-negative design for studies of the effectiveness of influenza vaccine. Vaccine. 2013; 31(30):3104–9. doi: 10.1016/j.vaccine.2013.04.026 .

Jackson ML, Nelson JC. The test-negative design for estimating influenza vaccine effectiveness. Vaccine. 2013; 31(17):2165–8. doi: 10.1016/j.vaccine.2013.02.053 .

Talbot HK, Nian H, Chen Q, Zhu Y, Edwards KM, Griffin MR. Evaluating the case-positive, control test-negative study design for influenza vaccine effectiveness for the frailty bias. Vaccine. 2016; 34(15):1806–9. doi: 10.1016/j.vaccine.2016.02.037 .

Download references

Acknowledgement

The authors wish to thank Dr. Ivo Foppa and two reviewers for helpful comment.

This research was supported by the National Institute of Allergies and Infectious Diseases of the National Institutes of Health (NIH) under Award R01AI110474, and by IPA 1110376-05 with the Centers for Disease Controls and Prevention (CDC). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or the CDC.

Availability of data and materials

Not applicable. No data or other materials are used in this work.

Author information

Authors and affiliations.

Department of Biostatistics and Bioinformatics, Rollins School of Public Health, Emory University, Atlanta, 30322, USA

Meng Shi, Qian An, Kylie E. C. Ainslie & Michael Haber

Division of Infectious Diseases, Department of Medicine, School of Medicine, Emory University, Atlanta, 30322, USA

Walter A. Orenstein

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Michael Haber .

Ethics declarations

Ethics approval and consent to participate.

Not applicable. No data on human subjects are used in this work.

Consent for publication

All the authors have read and approved the manuscript and agreed to be included in the publication.

Competing interests

The authors declares that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional information

Authors’ contribution.

MS, QA and MH developed the model and wrote the manuscript. WO and KA made significant revisions. All authors have read the manuscript and agreed to its content.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Shi, M., An, Q., Ainslie, K. et al. A comparison of the test-negative and the traditional case-control study designs for estimation of influenza vaccine effectiveness under nonrandom vaccination. BMC Infect Dis 17 , 757 (2017). https://doi.org/10.1186/s12879-017-2838-2

Download citation

Received : 19 January 2017

Accepted : 16 November 2017

Published : 08 December 2017

DOI : https://doi.org/10.1186/s12879-017-2838-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Case-control study
• Test-negative study
• Probability model

BMC Infectious Diseases

ISSN: 1471-2334

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Interim Estimates of Vaccine Effectiveness of Pfizer-BioNTech and Moderna COVID-19 Vaccines Among Health Care Personnel — 33 U.S. Sites, January–March 2021

Weekly / May 21, 2021 / 70(20);753–758

On May 14, 2021, this report was posted online as an MMWR Early Release.

Please note:  This report has been corrected.

Tamara Pilishvili, PhD 1 ; Katherine E. Fleming-Dutra, MD 1 ; Jennifer L. Farrar, MPH 1 ; Ryan Gierke, MPH 1 ; Nicholas M. Mohr, MD 2 ; David A. Talan, MD 3 ; Anusha Krishnadasan, PhD 3 ; Karisa K. Harland, PhD 2 ; Howard A. Smithline, MD 4 ; Peter C. Hou, MD 5 ; Lilly C. Lee, MD 6 ; Stephen C. Lim, MD 7 ; Gregory J. Moran, MD 3 ; Elizabeth Krebs, MD 8 ; Mark Steele, MD 9 ; David G. Beiser, MD 10 ; Brett Faine, PharmD 2 ; John P. Haran, MD, PhD 11 ; Utsav Nandi, MD 12 ; Walter A. Schrading, MD 13 ; Brian Chinnock, MD 14 ; Daniel J. Henning, MD 15 ; Frank LoVecchio, DO 16 ; Joelle Nadle, MPH 17 ; Devra Barter, MSc 18 ; Monica Brackney, MS 19 ; Amber Britton, MPH 20 ,21 ; Kaytlynn Marceaux-Galli, MPH 22 ; Sarah Lim, MBBCh 23 ; Erin C. Phipps, DVM 24 ,25 ; Ghinwa Dumyati, MD 26 ; Rebecca Pierce, PhD 27 ; Tiffanie M. Markus, PhD 28 ; Deverick J. Anderson, MD 29 ; Amanda K. Debes, PhD 30 ; Michael Lin, MD 31 ; Jeanmarie Mayer, MD 32 ; Hilary M. Babcock, MD 33 ; Nasia Safdar, MD, PhD 34 ,35 ; Marc Fischer, MD 1 ; Rosalyn Singleton, MD 36 ; Nora Chea, MD 1 ; Shelley S. Magill, MD, PhD 1 ; Jennifer Verani, MD 1 ; Stephanie Schrag, DPhil 1 ; Vaccine Effectiveness Among Healthcare Personnel Study Team ( View author affiliations )

What is already known about this topic?

Health care personnel (HCP) are at high risk for COVID-19. The early distribution of two mRNA COVID-19 vaccines (Pfizer-BioNTech and Moderna) to HCP provided an opportunity to examine vaccine effectiveness in a real-world setting.

What is added by this report?

The first U.S. multisite test-negative design vaccine effectiveness study among HCP found a single dose of Pfizer-BioNTech or Moderna COVID-19 vaccines to be 82% effective against symptomatic COVID-19 and 2 doses to be 94% effective.

What are the implications for public health practice?

The mRNA vaccines are highly effective at preventing symptomatic COVID-19 among U.S. HCP. High vaccination coverage among HCP and the general population is critical to prevent COVID-19 in the United States.

Views: Views equals page views plus PDF downloads

Throughout the COVID-19 pandemic, health care personnel (HCP) have been at high risk for exposure to SARS-CoV-2, the virus that causes COVID-19, through patient interactions and community exposure ( 1 ). The Advisory Committee on Immunization Practices recommended prioritization of HCP for COVID-19 vaccination to maintain provision of critical services and reduce spread of infection in health care settings ( 2 ). Early distribution of two mRNA COVID-19 vaccines (Pfizer-BioNTech and Moderna) to HCP allowed assessment of the effectiveness of these vaccines in a real-world setting. A test-negative case-control study is underway to evaluate mRNA COVID-19 vaccine effectiveness (VE) against symptomatic illness among HCP at 33 U.S. sites across 25 U.S. states. Interim analyses indicated that the VE of a single dose (measured 14 days after the first dose through 6 days after the second dose) was 82% (95% confidence interval [CI] = 74%–87%), adjusted for age, race/ethnicity, and underlying medical conditions. The adjusted VE of 2 doses (measured ≥7 days after the second dose) was 94% (95% CI = 87%–97%). VE of partial (1-dose) and complete (2-dose) vaccination in this population is comparable to that reported from clinical trials and recent observational studies, supporting the effectiveness of mRNA COVID-19 vaccines against symptomatic disease in adults, with strong 2-dose protection.

A test-negative design case-control study of mRNA COVID-19 VE is underway, with HCP being enrolled at 33 sites across 25 U.S. states; the planned interim analysis presented in this report includes data collected during January–March 2021.* A majority (75%) of enrolled HCP worked at acute care hospitals (including emergency departments), 25% worked in outpatient or specialty clinics, and <1% worked in long-term care facilities and urgent care clinics. HCP with the potential for exposure to SARS-CoV-2 through direct patient contact or for indirect exposure (e.g., through infectious materials) were eligible for enrollment. † Case-patients and control participants (controls) were identified through routine employee testing performed based on site-specific occupational health practices. HCP with a positive SARS-CoV-2 polymerase chain reaction (PCR) or antigen-based test result and at least one COVID-19–like illness symptom § were enrolled as case-patients, and HCP with a negative SARS-CoV-2 PCR test result, regardless of symptoms, were eligible for enrollment as controls. Controls were frequency matched to case-patients (aiming for a ratio of three controls per case-patient) by site and week of test. HCP who reported having received a positive SARS-CoV-2 PCR or antigen-based test result >60 days earlier (i.e., with a previous SARS-CoV-2 infection) were excluded. Information on demographics, COVID-19–like illness symptoms within 14 days before or after the testing date, and presence of underlying conditions and risk factors for severe COVID-19 ¶ were collected through HCP interviews or self-completed surveys. Medical records were reviewed to collect data on SARS-CoV-2 test dates, type, and results and on medical care sought for COVID-19–like illness. Vaccination records, including dates and type of COVID-19 vaccine received, were obtained from occupational health or other verified sources (e.g., vaccine card, state registry, or medical record).

HCP were defined as unvaccinated if they had not received any COVID-19 vaccine doses or had received their first dose after the test date. The interval of 0–13 days from receipt of the first dose was defined as the time before first dose vaccine effect. The effectiveness of a single dose was measured during the interval from 14 days after the first dose through 6 days after the second dose. Because of the potential for vaccine-related reactions to influence HCP testing behaviors, sensitivity analyses of single-dose VE were conducted 1) excluding participants tested within 0–2 days of receiving the second dose and 2) measuring VE before receiving the second dose. Effectiveness of 2 doses was measured ≥7 days after the receipt of the second dose, consistent with the Pfizer-BioNTech clinical trial procedure ( 3 ). Sensitivity analyses measuring 2-dose effectiveness ≥14 days after the second dose were conducted, consistent with the Moderna clinical trial procedure ( 4 ). Conditional logistic regression was used to estimate matched odds ratios (mORs) adjusted for age, race/ethnicity, and presence of underlying conditions. VE was estimated as 100% × (1–mOR) for 1 or 2 doses, compared with no doses. Because of the small sample size, analyses could not be stratified by COVID-19 vaccine type. All statistical analyses were conducted using SAS (version 9.4; SAS Institute). This activity was reviewed by CDC and was conducted consistent with applicable federal law and CDC policy.**

As of March 18, 2021, 623 case-patients and 1,220 controls had been enrolled. The median ages of case-patients and controls were 38 years (range = 19–69 years) and 37 years (range = 19–76 years), respectively ( Table 1 ). The majority of HCP (60% of case-patients and 64% of controls) worked in occupational categories with substantial anticipated direct patient contact and were aged 19–49 years (75% and 76%, respectively), female (84% and 82%, respectively), and non-Hispanic White (64% and 70%, respectively). Underlying conditions associated with increased risk for severe COVID-19 were reported by 77% of case-patients and 75% of controls. Case-patients were significantly more likely than controls to have fever (40% versus 23%, p<0.001), cough (56% versus 22%, p<0.001), or shortness of breath (26% versus 7%, p<0.001); 5% of case-patients and 14% of controls reported only mild symptoms (sore throat, headache, runny nose, or congestion; p<0.001); 17% of controls reported no symptoms. Only 12 (2%) case-patients and 10 (1%) controls had severe illness requiring hospitalization, and no deaths occurred in either group.

Ten percent of case-patients and 20% of controls had received 1 dose of COVID-19 vaccine ≥14 days before the test date, and 3% of case-patients and 15% of controls had received 2 doses ≥7 days before the test date ( Table 2 ). Among vaccinated persons, 76% of case-patients and 78% of controls received the Pfizer-BioNTech vaccine; the remainder received the Moderna vaccine. The adjusted single-dose VE was 82% (95% CI = 74%–87%) and was similar for both 1-dose sensitivity analyses (before dose 2: VE = 74%, 95% CI = 62%–82%; excluding days 0–2 after dose 2: VE = 78%, 95% CI = 68%–84%). The adjusted 2-dose VE was 94% (95% CI = 87%–97%); effectiveness ≥14 days after the second dose was similar (VE = 90%, 95% CI = 77%–96%).

This multisite test-negative design case-control study found that authorized mRNA COVID-19 vaccines (Pfizer-BioNTech and Moderna) are highly effective against symptomatic COVID-19 among HCP. Effectiveness of a complete 2-dose regimen of these vaccines was estimated to be 94%, consistent with findings from two clinical trials ( 3 , 4 ). Although the case definition applied in this study was broader than that used in both clinical trials ( 3 , 4 ), 93% and 88% of cases included in this study met the respective Pfizer-BioNTech and Moderna trial case definitions. The results are also consistent with findings from an observational study among the general adult population from Israel ( 5 ), two cohort studies among HCP from the United Kingdom, †† and recently reported interim results from a U.S. cohort evaluation among HCP and frontline workers ( 6 ).

Effectiveness of a single dose, estimated to be 82% in this report, has also been demonstrated in phase III trials and recent observational studies. The estimated effectiveness found in this report is higher than estimates of single-dose effectiveness found in the Pfizer-BioNTech clinical trial (efficacy 52%; 95% CI = 30%–68%) ( 3 ) and an observational study from Israel ( 5 ). In the Israeli study, the Pfizer-BioNTech VE against symptomatic illness among the general adult population was 57% (95% CI = 50%–63%) and 66% (95% CI = 57%–73%) measured during 14–20 and 21–27 days, respectively, after the first dose ( 5 ). These differences might be related to the younger age of the HCP population in this study (<2% of participants aged ≥65 years) compared with the age of the Israeli study population (13% aged ≥70 years). In two cohort studies among HCP, the single-dose effectiveness of the Pfizer-BioNTech vaccine was consistent with the estimates in this report, with 72% effectiveness (95% CI = 58%–86%) 21 days after the first dose in a U.K. study ( 7 ) and 80% effectiveness (95% CI = 59%–90%) ≥14 days after the first dose in a U.S. cohort study ( 6 ). Because the single-dose effectiveness estimates in this and other studies were based on a short follow-up, the duration of this level of protection from a single dose is unknown.

The findings in this report are subject to at least four limitations. First, testing for SARS-CoV-2 infection among HCP was based on occupational health practices at each facility, and no changes in routine testing practices were reported after vaccine introduction. If vaccinated HCP were less likely to obtain testing than unvaccinated HCP, the VE might have been underestimated. Alternatively, if postvaccination reactions increased the likelihood that vaccinated HCP would seek testing, the VE might have been overestimated. However, the sensitivity analysis excluding the interval of 0–2 days after receipt of dose 2, the interval during which most postvaccination reactions would be expected to occur, did not significantly change effectiveness estimates. Second, because of the limited sample size, effectiveness by vaccine product, presence of underlying medical conditions, and disease severity could not be estimated. In addition, because of limited statistical power, effectiveness estimates could not be adjusted for other potential confounders, such as use of personal protective equipment, occupational categories, or workplace or community exposures. Third, the VE estimates might not be generalizable to the U.S. adult population because racial/ethnic minority groups disproportionately affected by COVID-19 and who may have had higher exposure risks in the community were underrepresented in this population, and the overall HCP population was younger than the general U.S. adult population. However, the study’s geographic coverage was broad, representing the population of U.S. HCP, and vaccination data were obtained from multiple sources. Finally, although HCP with a known past acute SARS-CoV-2 infection were excluded, those whose previous infection was unknown could not be excluded. Data collection for this study is ongoing and will allow effectiveness to be evaluated by vaccine type and among HCP subgroups.

These interim results demonstrate that complete vaccination with authorized mRNA COVID-19 vaccines is highly effective in preventing symptomatic COVID-19 among HCP, supporting the results of phase III trials and additional accruing evidence in recent observational studies. Real-world VE data are critical to guiding evolving COVID-19 vaccine policy. In addition to adherence to recommended infection control and prevention practices, a critical component of controlling the U.S. COVID-19 pandemic and protecting HCP is ensuring high coverage with safe and effective COVID-19 vaccines.

Acknowledgments

The health care personnel and health care systems who agreed to participate in this study; Jasmine Varghese; Taniece Eure; Rebecca M. Alkis Ramirez; Gregory Blazek; Kavitha Pathmarajah; Kelli Wallace; Alysia Horcher; Allison Schuette; Karin Hoth; Amanda Higgins; Brianna M. DiFronzo; Karen Hopcia; Theresa M. Orechia; Alexander B. Hill; Gabrielle Donohoe; Lily R. Johnsky; Jordyn M. Fofi; Steven E. Miyawaki; Jenson J. Kaithamattam; Michelle Chung; Nikita A. Umale; Mohammad Adrian Hasdianda; Guruprasad Jambaulikar; Tala Teymour; Maria Davila; Suzette Fernandez; Joshua Tiao; Eva Gonzalez; Reynaldo Padilla; Cynthia Delgado; Madeleine Manahan; Melanie Potts; Jessica Kuo; Alyssa Fowlds; Zoe Speight; Laurie Kemble; Danielle Beckham; Lori Wilkerson; Geneatra Green; Rachel Marrs; Katherine Schneider; Cathy Fairfield; Fred Ullrich; Virginia Mangolds; Morgan Nelson; Abigail Lopes; Scott Pelletier; Gloria Essien; Rebekah Peacock; Alan Jones; Bhagyashri Navalkele; Savannah Vann; Andrea Williams; Brooke Park; Eugene Melvin; Joel Rodgers; Nivedita Patkar; Delissa Tidwell-Hand; Whitney Covington; Michael C. Kurz; Peter Poerzgen; Layla A. Anderson; Kyle A. Steinbock; Christine D. Crider; Jennifer Smith; Ethan Lindgren; Linda Frank; Deborah Godine; Christopher Czaja; Anastasia Edwards; Elisabeth Harrington; Catherine Emanuel; Melissa Kellogg; Paula Clogher; Vivian Leung; Maya Dennis; Anisa Linton; James Meek; Linda Niccolai; Sara Niesobecki; Gwen Oliver; Monica Farley; Melissa Tobin-D’angelo; Stepy Thomas; Amy Tunali; Ingrid Zambrano; Erica Hazra; Kelli Williams; Kara Goldstone; Meaghan Woody; Timothy Walsh; Shannon Ball; Tameka Browne; Bailey Evenson; Rebecca Perlmutter; Emilija Motivans; Gerit Wagner; Tobias Leuthner; Ashley Fell; Kathy Como-Sabetti; Richard Danila; Ruth Lynfield; Leslie Baken; Dana Essex; Marla Sievers; Sarah Shrum Davis; Yadira Salazar-Sanchez; Lezah Brown; Kristina Flores; Caroline Habrun; Cathleen Concannon; Jane A. Woods; Christine Hurley; Monika Samper; Gabriela Escutia; Judith A. Guzman-Cottrill; William Schaffner; Kathryn Billings; Melinda Eady; Danielle Ndi; Carol Epling; Kristen Said; Michael Yarrington; Michael Smith; Becky Smith; Christopher Polage; Rachel Addison; Alicia Nelson; Katherine Foy; Shaun Truelove; Lori Neihaus; Jayme Hughes; Annie Voskertchian; Mary Carl Froilan; Christine Fukuda; Jinal Makhija; Lahari Thotapalli; Brian Orleans; Morgan Millar; Tavis Huber; Matthew Doane; Kristina Stratford; Jacob Crook; Candace Haroldsen; Ling Yan; Jessica N. Zhang; Olivia G. Arter; Grace Yuan; Candace R. Miller; David McDonald; Caroline A. O’Neil; Jahnavi Bongu; Aurora Pop-Vicas; Joe Mcbride; Hannah Schopp; Amanda Young; Joe Perzynski; Michelle Zimbric; Fauzia Osman; Linda Stevens; Kelly Mitchell; Timothy Thomas; Matthew Hirschfeld; Brian Lefferts; Joseph Klejka; Donna Galbreath; James Tiesinga; Michael Bruce; Leisha Nolen; Wendy Petersen; Amy Swango Wilson; Vicki Vermillion; Brianna Smith; Scott LaBrie; Lauren Gillott; Shannon Williams; Madilyn Short; Christine Desnoyers; Lucinda Alexie; Evelyn Smith; Rachelle White; Marissa Friday.

Vaccine Effectiveness Among Healthcare Personnel Study Team

Anna Yousaf, CDC COVID-19 Response Team; Yunmi Chung, CDC COVID-19 Response Team; Jennifer Onukwube, CDC COVID-19 Response Team; Wei Xing, CDC COVID-19 Response Team; Bradley Clinansmith, University of Iowa, Iowa City, Iowa; Lisandra Uribe, Olive View—University of California Los Angeles Medical Center, Los Angeles, California; Kye E. Poronsky, Baystate Medical Center, Springfield, Massachusetts; Dean M. Hashimoto, Brigham and Women’s Hospital, Boston, Massachusetts; Monica Bahamon, Jackson Memorial Hospital, Miami, Florida; Michelle St. Romain, Louisiana State University, New Orleans, Louisiana; Efrat Kean, Thomas Jefferson University Hospital, Philadelphia, Pennsylvania; Amy Stubbs, Truman Medical Center, Kansas City, Missouri; Sara Roy, University of Chicago, Chicago, Illinois; Gregory Volturo, University of Massachusetts, Worcester, Massachusetts; James Galbraith, University of Mississippi, Jackson, Mississippi; James C. Crosby, University of Alabama at Birmingham, Birmingham, Alabama; Megan R. Fuentes, University of Washington, Seattle, Washington; Mary Mulrow, Valleywise Health Medical Center, Phoenix, Arizona; Jane Lee, California Emerging Infections Program, Oakland, California; Helen Johnston, Colorado Department of Public Health and Environment, Denver, Colorado; AmberJean Hansen, Connecticut Emerging Infections Program, Yale School of Public Health, New Haven, Connecticut;; Scott K. Fridkin, Georgia Emerging Infections Program, Atlanta, Georgia, and Emory University School of Medicine, Atlanta, Georgia; Lucy E. Wilson, Department of Emergency Health Services, University of Maryland, Baltimore County, Baltimore Maryland; Sara Lovett, Minnesota Emerging Infections Program, Minnesota Department of Health, St. Paul, Minnesota; Melissa Christian, University of New Mexico, Albuquerque, New Mexico, and New Mexico Emerging Infections Program, Santa Fe, New Mexico; Christopher Myers, University of Rochester Medical Center/New York State—Rochester Emerging Infections Program, Rochester, New York; Valerie L.S. Ocampo, Public Health Division, Oregon Health Authority, Portland, Oregon; Keipp Talbot, Vanderbilt University Medical Center, Nashville, Tennessee; Jessica Seidelman, Duke Center for Antimicrobial Stewardship and Infection Prevention, Duke University School of Medicine, Durham, North Carolina; Aaron M. Milstone, Johns Hopkins School of Public Health, Baltimore, Maryland; Mary Hayden, Department of Medicine, Rush University, Medical Center, Chicago, Illinois; Matthew Samore, University of Utah, VA Salt Lake City Health Care System, Salt Lake City, Utah; Jennie H. Kwon, Washington University School of Medicine, St. Louis, Missouri; Daniel Shirley, University of Wisconsin—Madison, Madison, Wisconsin; Denise Dillard, Southcentral Foundation, Anchorage, Alaska; Jennifer Dobson, Yukon-Kuskokwim Health Corporation, Bethel, Alaska

Corresponding author: Tamara Pilishvili, [email protected] .

1 CDC COVID-19 Response Team; 2 University of Iowa, Iowa City, Iowa; 3 Olive View–University of California Los Angeles Medical Center, Los Angeles, California; 4 Baystate Medical Center, Springfield, Massachusetts; 5 Brigham and Women’s Hospital, Boston, Massachusetts; 6 Jackson Memorial Hospital, Miami, Florida; 7 Louisiana State University, New Orleans, Louisiana; 8 Thomas Jefferson University Hospital, Philadelphia, Pennsylvania; 9 Truman Medical Center, Kansas City, Missouri; 10 University of Chicago, Chicago, Illinois; 11 University of Massachusetts, Worcester, Massachusetts; 12 University of Mississippi, Jackson, Mississippi; 13 University of Alabama at Birmingham, Birmingham, Alabama; 14 University of California San Francisco, Fresno, California; 15 University of Washington, Seattle, Washington; 16 Valleywise Health Medical Center, Phoenix, Arizona; 17 California Emerging Infections Program, Oakland, California; 18 Colorado Department of Public Health and Environment, Denver, Colorado; 19 Connecticut Emerging Infections Program, Yale School of Public Health, New Haven, Connecticut; 20 Georgia Emerging Infections Program, Atlanta, Georgia; 21 Emory University School of Medicine, Atlanta, Georgia; 22 Maryland Department of Health, Baltimore, Maryland; 23 Minnesota Emerging Infections Program, Minnesota Department of Health, St. Paul, Minnesota; 24 University of New Mexico, Albuquerque, New Mexico; 25 New Mexico Emerging Infections Program, Santa Fe, New Mexico; 26 University of Rochester Medical Center/New York State—Rochester Emerging Infections Program, Rochester, New York; 27 Public Health Division Oregon Health Authority, Portland, Oregon; 28 Vanderbilt University Medical Center, Nashville, Tennessee; 29 Duke Center for Antimicrobial Stewardship and Infection Prevention, Duke University School of Medicine, Durham, North Carolina; 30 Johns Hopkins School of Public Health, Baltimore, Maryland; 31 Department of Medicine, Rush University Medical Center, Chicago, Illinois; 32 University of Utah, VA Salt Lake City Health Care System, Salt Lake City, Utah; 33 Washington University School of Medicine, St. Louis, Missouri; 34 University of Wisconsin—Madison, Madison, Wisconsin; 35 William S. Middleton Memorial Veterans Hospital, Madison, Wisconsin; 36 Alaska Native Tribal Health Consortium, Anchorage, Alaska.

All authors have completed and submitted the International Committee of Medical Journal Editors form for disclosure of potential conflicts of interest. Deverick Anderson is the owner of Infection Control Education for Major Sports, LLC, and reports grants from the Agency for Healthcare Research and Quality and personal fees from UpToDate, outside the submitted work. Ghinwa Dumyati reports grants from Pfizer and personal fees from Roche Diagnostics. Gregory Moran reports grants from I-Mab Biopharma and BeiGene and personal fees from Light AI. Mark Steele reports personal fees from Light AI, during the conduct of the study. No other potential conflicts of interest were disclosed.

* https://www.cdc.gov/vaccines/covid-19/downloads/hcp-early-phase-protocol-508.pdf

† https://www.cdc.gov/coronavirus/2019-ncov/hcp/guidance-risk-assesment-hcp.html

§ Health care personnel are considered symptomatic if one or more of the following signs and symptoms are present 14 days before or after the test date: fever (documented ≥100.4°F [38.0°C] or subjective), chills, cough (dry or productive), shortness of breath, chest pain or tightness, fatigue or malaise, sore throat, headache, runny nose, congestion, muscle aches, nausea or vomiting, diarrhea, abdominal pain, altered sense of smell or taste, loss of appetite, or red or bruised toes or feet.

¶ Underlying conditions grouped based on CDC guidelines identifying conditions associated or potentially associated with risk for severe COVID-19 illness. https://www.cdc.gov/coronavirus/2019-ncov/need-extra-precautions/people-with-medical-conditions.html

** This investigation was defined as having met the requirements for public health surveillance as defined in 45 C.F.R. part 46.102(l)(2) 21 C.F.R. part 56; 42 U.S.C. Sect. 241(d); 5 U.S.C. Sect. 552a; 44 U.S.C. Sect. 3501 et seq.

†† https://doi.org/10.1101/2021.03.09.21253218 ; https://doi.org/10.1101/2021.03.11.21253275

• Hughes MM, Groenewold MR, Lessem SE, et al. Update: characteristics of health care personnel with COVID-19—United States, February 12–July 16, 2020. MMWR Morb Mortal Wkly Rep 2020;69:1364–8. https://doi.org/10.15585/mmwr.mm6938a3 PMID:32970661
• Dooling K, Marin M, Wallace M, et al. The Advisory Committee on Immunization Practices’ updated interim recommendation for allocation of COVID-19 vaccine—United States, December 2020. MMWR Morb Mortal Wkly Rep 2021;69:1657–60. https://doi.org/10.15585/mmwr.mm695152e2 PMID:33382671
• Polack FP, Thomas SJ, Kitchin N, et al.; C4591001 Clinical Trial Group. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. N Engl J Med 2020;383:2603–15. https://doi.org/10.1056/NEJMoa2034577 PMID:33301246
• Baden LR, El Sahly HM, Essink B, et al.; COVE Study Group. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N Engl J Med 2021;384:403–16. https://doi.org/10.1056/NEJMoa2035389 PMID:33378609
• Dagan N, Barda N, Kepten E, et al. BNT162b2 mRNA Covid-19 vaccine in a nationwide mass vaccination setting. N Engl J Med 2021;384:1412–23. https://doi.org/10.1056/NEJMoa2101765 PMID:33626250
• Thompson MG, Burgess JL, Naleway AL, et al. Interim estimates of vaccine effectiveness of BNT162b2 and mRNA-1273 COVID-19 vaccines in preventing SARS-CoV-2 infection among health care personnel, first responders, and other essential and frontline workers—eight U.S. locations, December 2020–March 2021. MMWR Morb Mortal Wkly Rep 2021;70:495–500. https://doi.org/10.15585/mmwr.mm7013e3 PMID:33793460
• Hall VJ, Foulkes S, Saei A, et al.; SIREN Study Group. COVID-19 vaccine coverage in health-care workers in England and effectiveness of BNT162b2 mRNA vaccine against infection (SIREN): a prospective, multicentre, cohort study. Lancet 2021;397:1725–35. https://doi.org/10.1016/S0140-6736(21)00790-X PMID:33901423

Abbreviations: HCP = health care personnel; PCR = polymerase chain reaction. * Case-patients: HCP who received positive SARS-CoV-2 PCR or antigen-based test results and had one or more symptoms of COVID-19–like illness; controls: HCP who received negative SARS-CoV-2 PCR test results. † Includes Asian or Pacific Islander (44 case-patients, 109 controls), American Indian or Alaska Native (23 case-patients, 35 controls), multiple races (5 case-patients, 19 controls), and missing race (5 case-patients, 16 controls). § Substantial patient contact occupational categories: health care providers (physicians, residents, fellows, attending physicians, nurse practitioners, and physician assistants), nurses (registered nurses, other nursing providers including intensive care unit nurses, nurse managers, and midwives), direct patient assistants (licensed practical nurses, certified nursing assistants, patient care technicians and assistants, medical assistants, COVID-19 testers, phlebotomists, home health care providers, emergency medical services providers, and paramedics), and medical therapists (physical therapists; physical therapy assistants; rehabilitation providers; rehabilitation aides; occupational therapists; speech and language pathologists; respiratory therapists; radiology technicians; dental health care providers, including dentists or dental hygienists; and surgical, medical, or emergency technicians). ¶ Moderate patient contact occupational categories: behavioral/social services providers (behavioral health providers [excluding physician psychiatrists], chaplains, social workers and assistants, care coordinators, interpreters, patient registration personnel, health educators, genetic counselors, ambulance dispatchers, dieticians, and research staff members), and environmental services providers (facilities staff members, food services workers, transport workers, patient transport workers, and drivers). ** Minimal patient contact occupational categories: administrative or ward clerks, symptom checkers, telehealth trainers, clinical support staff members, equipment and sterile processing technicians, medical equipment sales personnel, laboratory personnel, and pharmacists. †† Undefined patient contact occupational categories: others who could not be classified into any of the preceding categories and those with missing information. §§ Conditions associated with definite or potential increased risk for severe COVID-19 illness as defined by CDC. https://www.cdc.gov/coronavirus/2019-ncov/need-extra-precautions/people-with-medical-conditions.html?CDC_AA_refVal=https%3A%2F%2Fwww.cdc.gov%2Fcoronavirus%2F2019-ncov%2Fneed-extra-precautions%2Fgroups-at-higher-risk.html ¶¶ Among HCP who reported diabetes mellitus, no case-patients and two controls (<1% of all controls) reported type 1 diabetes, eight case-patients (1% of all case-patients) and nine controls (<1% of all controls) reported type 2 diabetes, and 20 case-patients (3%) and 46 controls (4%) did not specify a diabetes type. *** Immunocompromising conditions include immunosuppression medication (e.g., corticosteroids, chemotherapy, or other immunosuppressive medications), solid organ transplant, hematopoietic stem cell transplant, HIV, thalassemia, or active cancer (current cancer or in treatment or received diagnosis within last 12 months). ††† Smoking includes cigarettes, tobacco, e-cigarettes/vaping, or marijuana use. §§§ Statistically significant difference between case-patients and controls; chi-square test, p-value<0.001. ¶¶¶ Other symptoms include chest pain or tightness, abdominal pain, loss of appetite, red or bruised toes or feet, headache, runny nose, or congestion. **** One person’s first dose was Moderna vaccine and second dose was Pfizer-BioNTech vaccine.

Abbreviations: CI = confidence interval; HCP = health care personnel; mOR = matched odds ratio; OR = odds ratio; PCR = polymerase chain reaction; VE = vaccine effectiveness. * Case-patients: HCP who received positive SARS-CoV-2 PCR or antigen-based test results and had one or more symptoms of COVID-19–like illness; controls: HCP who received negative SARS-CoV-2 PCR test results. † VE (Pfizer-BioNTech and Moderna) was estimated using a conditional logistic regression model accounting for matching by site of enrollment and week of test date. § OR used in conditional logistic regression model to calculate VE was adjusted for age, race, and presence of underlying conditions: VE = 100% × (1−mOR).

Suggested citation for this article: Pilishvili T, Fleming-Dutra KE, Farrar JL, et al. Interim Estimates of Vaccine Effectiveness of Pfizer-BioNTech and Moderna COVID-19 Vaccines Among Health Care Personnel — 33 U.S. Sites, January–March 2021. MMWR Morb Mortal Wkly Rep 2021;70:753–758. DOI: http://dx.doi.org/10.15585/mmwr.mm7020e2 .

MMWR and Morbidity and Mortality Weekly Report are service marks of the U.S. Department of Health and Human Services. Use of trade names and commercial sources is for identification only and does not imply endorsement by the U.S. Department of Health and Human Services. References to non-CDC sites on the Internet are provided as a service to MMWR readers and do not constitute or imply endorsement of these organizations or their programs by CDC or the U.S. Department of Health and Human Services. CDC is not responsible for the content of pages found at these sites. URL addresses listed in MMWR were current as of the date of publication.

All HTML versions of MMWR articles are generated from final proofs through an automated process. This conversion might result in character translation or format errors in the HTML version. Users are referred to the electronic PDF version ( https://www.cdc.gov/mmwr ) and/or the original MMWR paper copy for printable versions of official text, figures, and tables.

Exit Notification / Disclaimer Policy

• The Centers for Disease Control and Prevention (CDC) cannot attest to the accuracy of a non-federal website.
• Linking to a non-federal website does not constitute an endorsement by CDC or any of its employees of the sponsors or the information and products presented on the website.
• You will be subject to the destination website's privacy policy when you follow the link.
• CDC is not responsible for Section 508 compliance (accessibility) on other federal or private website.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 14 January 2022

Effectiveness of COVID-19 booster vaccines against COVID-19-related symptoms, hospitalization and death in England

• Nick Andrews 1 , 2   na1 ,
• Julia Stowe 1 , 2   na1 ,
• Freja Kirsebom   ORCID: orcid.org/0000-0003-4585-319X 1 ,
• Samuel Toffa 1 ,
• Ruchira Sachdeva 1 ,
• Charlotte Gower 1 ,
• Mary Ramsay 1 , 2 &
• Jamie Lopez Bernal   ORCID: orcid.org/0000-0002-1301-5653 1 , 2 , 3  

Nature Medicine volume  28 ,  pages 831–837 ( 2022 ) Cite this article

73k Accesses

234 Citations

154 Altmetric

Metrics details

• Epidemiology
• Infectious diseases

Booster vaccination with messenger RNA (mRNA) vaccines has been offered to adults in England starting on 14 September 2021. We used a test-negative case–control design to estimate the relative effectiveness of a booster dose of BNT162b2 (Pfizer-BioNTech) compared to only a two-dose primary course (at least 175 days after the second dose) or unvaccinated individuals from 13 September 2021 to 5 December 2021, when Delta variant was dominant in circulation. Outcomes were symptomatic coronavirus disease 2019 (COVID-19) and hospitalization. The relative effectiveness against symptomatic disease 14–34 days after a BNT162b2 or mRNA-1273 (Moderna) booster after a ChAdOx1-S (AstraZeneca) and BNT162b2 as a primary course ranged from around 85% to 95%. Absolute vaccine effectiveness ranged from 94% to 97% and was similar in all age groups. Limited waning was seen 10 or more weeks after the booster. Against hospitalization or death, absolute effectiveness of a BNT162b2 booster ranged from around 97% to 99% in all age groups irrespective of the primary course, with no evidence of waning up to 10 weeks. This study provides real-world evidence of substantially increased protection from the booster vaccine dose against mild and severe disease irrespective of the primary course.

Similar content being viewed by others

Effectiveness of a second BNT162b2 booster vaccine against hospitalization and death from COVID-19 in adults aged over 60 years

Vaccine effectiveness of heterologous CoronaVac plus BNT162b2 in Brazil

Duration of protection of CoronaVac plus heterologous BNT162b2 booster in the Omicron period in Brazil

Real-world effectiveness data has demonstrated high levels of short-term protection by COVID-19 vaccines against clinical disease and, more so, against severe outcomes, including hospitalization and death 1 , 2 , 3 , 4 , 5 , 6 , 7 . Nevertheless, there is evidence that protection against symptomatic disease wanes over time 8 , 9 . Booster doses have now been implemented in the United Kingdom and elsewhere in order to combat the rise in COVID-19 cases and the additional threat of the winter 2021 influenza season.

We recently reported that vaccine effectiveness against symptomatic disease peaked in the early weeks after the second dose and then fell to 47.3 (95% confidence interval (CI), 45–49.6) and 69.7 (95% CI, 68.7–70.5) by ≥20 weeks against the Delta variant for ChAdOx1-S (AstraZeneca) and BNT162b (Pfizer-BioNTech)), respectively. Vaccine effectiveness against severe disease outcomes remained high up to 20 weeks after vaccination in most groups; nevertheless, greater waning was seen in older adults and those with underlying medical conditions compared to young, healthy adults 8 .

In the United Kingdom, COVID-19 booster vaccines were introduced on 14 September 2021. Using evidence from the COV-BOOST trial, which demonstrated that the mRNA vaccines provide a strong booster effect with low reactogenicity, regardless of the vaccine given in the primary course, the UK Joint Committee on Vaccination and Immunisation recommended either a BNT162b2 or a half dose (50 µg) of mRNA-1273 (Moderna) vaccine to be given as a booster dose no earlier than 6 months after completion of the primary vaccine course 10 , 11 . In this initial phase of the UK booster program, the following groups were eligible: all adults >50 years and those 16–49 years with underlying health conditions that put them at higher risk of severe COVID-19, adult carers and adult household contacts (aged ≥16 years) of immunosuppressed individuals and healthcare workers.

In this study, we aimed to estimate the effectiveness of the BNT162b2 and mRNA-1273 booster vaccines against symptomatic disease, hospitalization and death in adults in England. Table 1 outlines the main findings and implications for policy of our study.

Descriptive statistics and characteristics

From 13 September 2021 to 5 December 2021, there were a total of 893,845 eligible test results for individuals aged ≥18 years with a test date within 10 days of their symptom onset date and a link to the National Immunisation Management System (NIMS), with a 91.04% match rate. Of these eligible participants, 278,096 (31.1%) were unvaccinated, 223,198 received ChAdOx1-S 175 days after a second dose and 171,079 received BNT162b2 175 days after a second dose. Of those who had received a booster dose, 89,019 received a ChAdOx1-S primary course and 132,453 received a BNT162b2 primary course. Of the 343,955 positive cases included in the analysis, 4,377 (1.27%) were hospitalized for any reason (excluding injuries) within 14 days of the test. A description of the eligible tests is given in Supplementary Table 1 .

Vaccine effectiveness for symptomatic disease

An overall effect on the proportion of cases and controls was seen from around day 7 after the booster dose and stabilized at day 11 (Extended Data Fig. 1 ). In individuals aged 18 to 49 years where the primary course was ChAdOx1-S vaccine, relative to those who had received only two doses, effectiveness against symptomatic disease peaked at 14–35 days after the BNT162b2 booster at 89.6% (95% CI, 88.6–90.4) and 95.3% (95% CI, 91.8–97.3) after the mRNA-1273 booster (Table 2 and Fig. 1 ). In individuals where BNT162b2 was the primary course, relative vaccine effectiveness 14-34 days a BNT162b2 booster was 82.8% (81.8-83.7) and after a mRNA-1273 booster 90.9% (84.5–94.7). Relative vaccine effectiveness with the BNT162b2 booster decreased slightly in the 35- to 69-day and ≥70-day periods (later follow-up was not available for mRNA-1273). The same analysis in individuals aged 50 years and older gave similar results (Table 2 and Fig. 1 ).

a , b , Vaccine effectiveness estimates (95% CI) against symptomatic disease in time intervals after booster according to primary course in individuals aged 18–49 years ( a ) 50 years and older ( b ). Dose 2 was received at 175 days as the baseline.

In the secondary analysis, which used the 2- to 6-day period after the booster dose as the baseline, results were similar to the primary analysis (Table 2 and Extended Data Fig. 2 ). In the analysis using the unvaccinated individuals as the baseline, the booster dose was associated with an absolute vaccine effectiveness from 14 to 34 days after a BNT162b2 booster of 94.4% (95% CI, 94.1–94.7) following either a ChAdOx1-S or BNT162b2 primary course in individuals 50 years and older. With an mRNA-1273 booster, absolute vaccine effectiveness was 97.0 (95% CI, 96.0–97.8) after a ChAdOx1-S primary course and 94.8% (95% CI, 92.7–96.3) after a BNT162b2 primary course (Table 3 and Extended Data Fig. 3 ).

Vaccine effectiveness for hospitalization and death

High levels of protection were also seen against hospitalization in both age groups. In individuals aged 50 years and older, the vaccine effectiveness 14-34 days after a BNT162b2 booster dose, relative to unvaccinated individuals, was 99.2% (95% CI, 98.6–99.5) when the primary course was ChAdOx1-S and 98.6% (95% CI, 98.0–99.0) when BNT162b2 was used as the primary course.

A similarly high level of protection was seen in the younger age group, with a vaccine effectiveness estimate of 97.5% (95% CI, 93.3–99.1) when the primary course was ChAdOx1-S and 98.8% (95% CI, 97.2–99.5) when BNT162b2 was used as the primary course (Table 3 and Fig. 2 ). There was little evidence of any waning in vaccine effectiveness against hospitalization up to 69 days after the booster.

a – c , Vaccine effectiveness estimates (95% CI) in time intervals after booster according to primary course against hospitalization in individuals aged 18–49 years ( a ) and 50 years and older ( b ) and against death in individuals aged 50 years and older ( c ). Unvaccinated individuals served as the baseline.

Vaccine effectiveness against death in individuals 50 years and older 14–34 days after a BNT162b2 booster dose relative to the unvaccinated was 97.8 (95% CI, 94.4–99.1) after a ChAdOx1-S primary course and 98.7% (95% CI, 97.4–99.4) when the primary course was BNT162b2 (Table 4 and Fig. 2 )

Interval between dose 2 and the booster dose

After assessing the distribution of intervals between dose 2 and the booster dose for cases and controls by age group and manufacturer, the interval between dose 2 and the booster was split into three periods: 25–29, 30–34 and 35 or more weeks (Extended Data Fig. 4 ). Due to the roll out of the vaccine program, there were more individuals who had received a second dose of BNT162b2 at an earlier time point; therefore, the majority of the individuals who had the longest interval between dose 2 and the booster had a BNT162b2 primary course. Analyses by interval between dose 2 and dose 3 were thus restricted to those who received BNT162b2 as the primary course.

A shorter interval between dose 2 and the booster of 25–29 weeks compared to the baseline interval of 35 weeks or more was associated with an increased adjusted odds ratio of 1.54 (95% CI, 1.35–1.76) for becoming a symptomatic case. This was also seen in the 30- to 34-week interval, with an adjusted odds ratio of 1.32 (1.12–1.56). Although remaining high, the adjusted vaccine effectiveness estimates decreased from 95.6% (95% CI, 94.9–96.1) in the 35 weeks or more interval to 93.2% (95% CI, 92.8–93.6) in the shortest interval between dose 2 and the booster (Supplementary Table 2 ). A test for the interaction effect of age was not significant ( P = 0.15).

This study provides evidence of a substantial increase in protection against symptomatic COVID-19 disease after a booster dose of BNT162b2 or mRNA-1273 vaccine during the period when the Delta variant was the dominant strain in the United Kingdom. Very high levels of protection were seen against hospitalization or death with a BNT162b2 booster. Vaccine effectiveness of a booster dose was very similar irrespective of the vaccine used in the primary course. A longer interval between dose 2 and the booster doses was associated with small improvements in vaccine effectiveness 12 .

These findings suggest that the booster offers very high levels of protection against mild and severe disease. Although a small amount of waning in protection against symptomatic disease is seen from 10 weeks after the booster, there is no clear evidence of waning against severe disease up to 10 weeks after the booster. Given the recent deployment of the booster program in the United Kingdom, further follow-up is needed to understand how protection changes longer term against both mild and severe disease. The slightly lower relative VE estimates of the booster in individuals with BNT162b2 as a primary course compared to the ChAdOx1-S in the primary analysis is due to the different baseline with higher VE after two doses of BNT162b2 as compared to ChAdOx1-S (ref. 8 ). When using unvaccinated controls, there was little difference in observed vaccine effectiveness of the booster dose with either primary course. We also observed a peak in testing at day 1 after the booster dose, which is likely to be reactogenicity effects so shortly after the vaccine, as has been reported previously 13 . The improved vaccine effectiveness with a longer interval between dose 2 and the booster suggests that there will be some benefit in delaying booster doses. Nevertheless, this improvement was only small and has to be balanced with the reduced protection among those who have received just two doses (where protection may have waned), compared to protection from the booster even with a relatively short interval. This finding was also similar to the reduced effectiveness among those who had a shorter interval between doses 1 and 2 (refs. 8 , 14 ). Furthermore, similar findings are also seen with history of prior infection, whereby a longer interval between infection and vaccination was associated with increased protection 15 .

In Israel, a booster program began in July 2021. Bar-On et al. reported an adjusted rate ratio of 11.3 (10.4–12.3) against confirmed infection in booster dose recipients compared to those who received only two doses (equivalent to relative vaccine effectiveness of 91.2%) 16 . This is slightly higher than the relative vaccine effectiveness that we report, which could reflect lower two-dose vaccine effectiveness in the comparison group in Israel, where a greater degree of waning has previously been reported 9 , 17 , 18 . Even greater protection has been reported in Israel against severe disease 16 , 19 . We were unable to find other studies reporting vaccine effectiveness of a third dose when ChAOx1-S was used as the primary course or when mRNA-1273 was used as the booster.

This is an observational study with a number of possible biases and should be interpreted with caution. The imperfect sensitivity of PCR testing could cause misclassification of both cases and controls in a test-negative case–control analysis, which could attenuate vaccine effectiveness estimates. Many individuals will also have been previously infected, so the vaccine effectiveness measured is in the context of a population in which many might have already had natural exposure. We adjust for measured confounders, but there may be residual confounding that we could not account for. Nevertheless, the similarity of the vaccine effectiveness estimates using those with two doses and no booster as the baseline and using the 2- to 6-day period after booster as the baseline suggests that residual confounding is small. Use of individuals who are unvaccinated as a comparator to obtain absolute effectiveness is most susceptible to residual confounding, as the unvaccinated population may differ in many ways from those who have had vaccine doses, which could lead to underestimation of vaccine effectiveness 8 . Despite this potential underestimation, using the unvaccinated comparator, the absolute vaccine effectiveness estimates were over 93%. Due to small numbers, we were only able to report the early effects of the booster program, and it is not yet clear how long protection against COVID-19 following booster vaccination will last.

For the analysis by interval between dose 2 and dose 3, it should be noted that those who had a longer interval between dose 2 and dose 3 are likely to have had a shorter interval between dose 1 and dose 2. As these variables are colinear, it is not possible to adjust for interval between dose 1 and dose 2. In these analyses, we were unable to report on the half dose (50 µg) of mRNA-1273 vaccine due to low numbers, as the majority of booster doses given in this period were BNT162b2. We were unable to assess the vaccine effectiveness in all those targeted for a booster dose, such as individuals with underlying health conditions, adult carers and adult household contacts of immunosuppressed individuals, due to small numbers and difficultly identifying these individuals with the dataset.

Our study provides real-world evidence of substantially increased protection from the booster dose against symptomatic disease and hospitalization in those aged ≥50 years irrespective of which primary course was received. This finding indicates that a high level of protection was achieved among older adults, who are more vulnerable to severe COVID-19. This protection will be important in the 2021–2022 winter period, when COVID-19 is likely to co-circulate alongside other respiratory viruses, including seasonal influenza virus.

Study design

We used a test-negative case–control design to estimate vaccine effectiveness of a booster dose of BNT162b2 vaccine against PCR-confirmed symptomatic disease and hospitalization. We compared vaccination status in symptomatic adults ≥18 years with PCR-confirmed severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection with the vaccination status in individuals who reported symptoms but had a negative SARS-CoV-2 PCR test result. Because an mRNA-1273 vaccine, as a primary course, was not made available until later in the vaccine program, insufficient time had elapsed for a booster dose to be indicated in this group. The study protocol is available in the Supplementary Appendix .

Ethical approval

Surveillance of COVID-19 testing and vaccination was undertaken under Regulation 3 of The Health Service (Control of Patient Information) Regulations 2002 to collect confidential patient information ( www.legislation.gov.uk/uksi/2002/1438/regulation/3/ made) under Sections 3(i) (a) to (c), 3(i)(d) (i) and (ii) and 3(3). The study protocol was subject to an internal review by the Public Health England Research Ethics and Governance Group and was found to be fully compliant with all regulatory requirements. As no regulatory issues were identified and ethical review is not a requirement for this type of work, it was decided that a full ethical review would not be necessary.

Data sources

Vaccination data.

NIMS contains demographic information on the entire population of England for individuals who are registered with a general practitioner in England, and this information is used to record all COVID-19 vaccinations. These data were accessed on 14 December 2021. The information used from NIMS included all dates of COVID-19 vaccination and vaccine manufacturer for each dose. Demographic data such as sex, date of birth, ethnicity and residential address were extracted. Addresses were used to determine index of multiple deprivation quintile and were also linked to Care Quality Commission-registered care homes using the unique property reference number. NIMS also contained data on geography (National Health Service (NHS) region), risk groups status, clinically extremely vulnerable and health/social care worker.

Booster doses were identified as being a third dose 175 days or more after a second dose and given after 13 September 2021. Individuals with four or more doses of vaccine, a mix of vaccines in their primary schedule or less than 19 days between their first and second dose were excluded.

COVID-19 testing data

SARS-CoV-2 PCR testing in the United Kingdom is undertaken by hospital and public health laboratories, as well as by community testing with the use of drive through or at-home testing, which is available to anyone with symptoms consistent with COVID-19, contacts of a confirmed case, care home staff and residents or anyone who has self-tested positive using a lateral flow device. Initially, data on all positive and negative test results for the period 8 December 2020 to 5 December 2021 were extracted for individuals aged ≥18 years (as of 31 August 2021). Any negative results of tests taken within 7 days of a previous negative test result, or where symptoms were recorded, with symptoms within 10 days of symptoms for a previous negative test result were dropped, as these results likely represent the same episode. Negative test results taken within 21 days before a positive test result were also excluded, as these results are likely to be false negative. Positive and negative test results within 90 days of a previous positive test result were also excluded. Participants contributed a maximum of one randomly chosen negative test result in the follow-up period. Data were restricted to persons who had reported symptoms and gave an onset date. Only persons who had undergone testing within 10 days after symptom onset were included in order to account for reduced sensitivity of PCR testing beyond this period. A small number of positive samples for which sequencing was done and they were found not to be the Delta variant were excluded. Finally, only samples taken from 13 September 2021 (week 37, 2021) were retained for analysis.

Testing data were linked to NIMS on 14 December 2021 using a combination of NHS number (a unique identifier for each person receiving medical care in the United Kingdom), date of birth, surname, first name and postcode using deterministic linkage with >95.5% uniqueness. The NIMS denominator file included information on potential confounding variables related to targeted populations.

Hospitalizations

Testing data were linked to the Emergency Care Dataset to assess vaccine effectiveness against hospitalization using data extracted on 15 December 2021. We included emergency care attendances among symptomatic cases within 14 days of the positive test result, which were not injury related and resulted in an inpatient admission. The Emergency Care Dataset includes hospital admissions through NHS emergency departments in England, but not elective admissions. Only first attendances in the 14-day period were selected if a person had more than one admission for emergency care. Data were extracted on 15 December 2021, with cases included if tested by 26 November 2021 to allow for lags in hospitalization.

Data management and linkage were carried out using Microsoft SQL Server Management Studio 18.

Statistical analysis

Analysis was by logistic regression with the PCR test result as the dependent variable, where those testing positive were cases and those testing negative were controls. Vaccination status was included as an independent variable and effectiveness defined as 1 odds of vaccination in cases/odds of vaccination in controls.

Vaccine effectiveness was adjusted in logistic regression models for age (5-year bands), sex, index of multiple deprivation (quintile), ethnic group, care-home residence status, geographic region (NHS region), period (calendar week of onset), health and social care worker status, clinical risk group status, clinically extremely vulnerable, severely immunosuppressed and previously testing positive. These factors were all considered potential confounders and thus included in all models. To assess the importance of previously testing positive, a sensitivity analysis was done excluding those previously testing positive for the comparison of the booster vaccine to unvaccinated.

Analyses were stratified by which primary doses had been received (ChAdOx1-S or BNT162b2), and any mixed primary courses were excluded.

Vaccine effectiveness against symptomatic disease was assessed for each primary course of vaccine in intervals in days after the booster dose. The BNT162b2 and mRNA-1273 booster doses were combined in the 0- to 1-day and 2- to 6-day postbooster periods. Subsequent periods (7–13, 14–34, 35–69 and ≥70 days) were analyzed separately. There were insufficient data in the last two periods for the mRNA-1273 booster.

Vaccine effectiveness against hospitalization and death was assessed in the combined 0–1 and 2–6 days after the booster and in the 7–13, 14–34 and 35–69 days after the BNT162b2 booster vaccine. All analysis were stratified by 18–49 years and ≥50 years, apart from the death analysis, which was only reported for individuals 50 years and older due to small numbers in <50 year olds.

In the primary analysis, those who had received the booster were compared to individuals who had received two primary doses with at least 175 days before the onset but with no booster dose recorded. In secondary analyses, we also compared to individuals who were completely unvaccinated and the 2- to 6-day period after the booster was received. The 2- to 6-day period was selected after plotting the data on case and control numbers after the booster dose and to avoid days 0 and 1 after the booster, when vaccine reactogenicity may affect the case–control ratio (Fig. 1 ). The analyses comparing to two doses or the 2- to 6-day postbooster period measured relative effectiveness to two doses, whereas the comparison to individuals who were unvaccinated measured absolute effectiveness of two doses and a booster. In the analysis comparing to individuals who were unvaccinated, we also assessed the remaining effectiveness of two doses at least 175 days (25 weeks) after the second dose.

Among individuals who received BNT162b2 as their primary course, an additional analysis was undertaken estimating the odds of testing positive in shorter intervals between dose 2 and the booster (25–29 and 30–34 weeks) relative to the longest interval (35 or more weeks). A test for the interaction effect of age was also performed. Vaccine effectiveness compared to individuals who were unvaccinated was also stratified by the interval between dose 2 and the booster.

Reporting Summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Data availability

No additional data available. Data cannot be made publicly available for ethical and legal reasons, that is public availability would compromise patient confidentiality as data tables list single counts of individuals rather than aggregated data. Databases used in this study include NIMS, Unified Sample Database and the Emergency Care Dataset.

Code availability

Model fitting code is available on our GitHub site ( https://github.com/UKHSAGithub/StataCode.git ).

Lopez Bernal, J. et al. Effectiveness of COVID-19 vaccines against the B.1.617.2 (Delta) variant. N. Engl. J. Med. 385 , 585–594 (2021).

Article   PubMed   Google Scholar  

Ismail, S. A. et al. Effectiveness of BNT162b2 mRNA and ChAdOx1 adenovirus vector COVID-19 vaccines on risk of hospitalisation among older adults in England: an observational study using surveillance data. Lancet Infect. Dis. 21 , 1539–1548 (2021).

Article   Google Scholar  

Vasileiou, E. et al. Interim findings from first-dose mass COVID-19 vaccination roll-out and COVID-19 hospital admissions in Scotland: a national prospective cohort study. Lancet 397 , 1646–1657 (2021).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Pritchard, E. et al. Impact of vaccination on SARS-CoV-2 cases in the community: a population-based study using the UK’s COVID-19 infection survey. Nat. Med. 27 , 370–1378 (2021).

Hyams, C. et al. Effectiveness of BNT162b2 and ChAdOx1 nCoV-19 COVID-19 vaccination at preventing hospitalisations in people aged at least 80 years: a test-negative, case-control study. Lancet Infect. Dis. 21 , 1539–1548 (2021).

Lopez Bernal, J. et al. Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines on covid-19 related symptoms, hospital admissions, and mortality in older adults in England: test negative case-control study. BMJ. 373 , n1088 (2021).

Pouwels, K. B. et al. Impact of Delta on viral burden and vaccine effectiveness against new SARS-CoV-2 infections in the UK. Nat. Med. 27 , 2127–2135 (2021).

Andrews, N. et al. Duration of protection against mild and severe disease by Covid-19 vaccines. N. Engl. J. Med. 386 , 340–350 (2022).

Article   CAS   PubMed   Google Scholar  

Goldberg, Y. et al. Waning immunity of the BNT162b2 vaccine: a nationwide study from Israel. N. Engl. J. Med. 385 , e85 (2021).

Alasdair, P. S. et al. Safety and immunogenicity of seven COVID-19 vaccines as a third dose (booster) following two doses of ChAdOx1 nCov-19 or BNT162b2 in the UK (COV-BOOST): a blinded, multicentre, randomised, controlled, phase 2 trial. Lancet 398 , 2258–2276 (2021).

JCVI statement regarding a COVID-19 booster vaccine programme for winter 2021 to 2022 (Department of Health and Social Care, 2021).

Arbel, R. et al. BNT162b2 vaccine booster and mortality due to COVID-19. N. Engl. J. Med. 385 , 2413–2420 (2021).

Amirthalingam, G. et al. Serological responses and vaccine effectiveness for extended COVID-19 vaccine schedules in England. Nat. Commun. 12 , 7217 (2021).

Abu-Raddad, L. J. et al. Association of prior SARS-CoV-2 infection with risk of breakthrough infection following mRNA vaccination in Qatar. JAMA . 326 , 1930–1939 (2021).

Bar-On, Y. M. et al. Protection of BNT162b2 vaccine booster against COVID-19 in Israel. N. Engl. J. Med. 385 , 1393–1400 (2021).

Israel, A. et al. Elapsed time since BNT162b2 vaccine and risk of SARS-CoV-2 infection: test negative design study. BMJ 375 , e067873 (2021).

Mizrahi, B. et al. Correlation of SARS-CoV-2-breakthrough infections to time-from-vaccine. Nat. Commun. 12 , 6379 (2021).

Barda, N. et al. Effectiveness of a third dose of the BNT162b2 mRNA COVID-19 vaccine for preventing severe outcomes in Israel: an observational study. Lancet 398 , 2093–2100 (2021).

Download references

Acknowledgements

We thank the UK Health Security Agency (UKHSA) COVID-19 Data Science Team, UKHSA Outbreak Surveillance Team, NHS England, NHS Digital and NHS Test and Trace for their roles in developing and managing the COVID-19 testing, variant identification and vaccination systems and datasets as well as reporting NHS vaccinators, NHS laboratories, UKHSA laboratories and lighthouse laboratories. We also thank the Wellcome Sanger Institute and other laboratories involved in whole-genome sequencing of COVID-19 samples, as well as the Joint Committee on Vaccination and Immunisation for advice and feedback in developing this study.

Author information

These authors contributed equally: Nick Andrews, Julia Stowe.

Authors and Affiliations

UK Health Security Agency, London, UK

Nick Andrews, Julia Stowe, Freja Kirsebom, Samuel Toffa, Ruchira Sachdeva, Charlotte Gower, Mary Ramsay & Jamie Lopez Bernal

NIHR Health Protection Research Unit in Vaccines and Immunisation, London School of Hygiene and Tropical Medicine, London, UK

Nick Andrews, Julia Stowe, Mary Ramsay & Jamie Lopez Bernal

NIHR Health Protection Research Unit in Respiratory Infections, Imperial College London, London, UK

Jamie Lopez Bernal

You can also search for this author in PubMed   Google Scholar

Contributions

J.L.B., N.A. and M.R. designed the study and developed the protocol and analysis plan. N.A., F.K. and J.S. cleaned and analyzed the data. J.S. drafted the manuscript. All authors contributed to the study design and revised the manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. J.L.B. and M.R. are the guarantors.

Corresponding author

Correspondence to Jamie Lopez Bernal .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Medicine thanks Susan Cheng, Daniel Altmann and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Jennifer Sargent was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended data fig. 1.

Distribution (%) and counts of cases and controls by interval from booster to onset.

Extended Data Fig. 2

Vaccine Effectiveness estimates (95% CI) against symptomatic disease in time intervals post booster according to primary course in individuals aged a) 18 to 49 years b) 50 years and over: Dose 3 at 2-6 days as baseline.

Extended Data Fig. 3

Vaccine Effectiveness estimates (95% CI) against symptomatic disease in time intervals post booster according to primary course in individuals aged a) 18 to 49 years b) 50 years and over: Unvaccinated baseline.

Extended Data Fig. 4

The distribution of intervals between dose 2 and the booster dose for cases and controls by age group and manufacturer.

Supplementary information

Supplementary information.

Supplementary Tables 1 and 2 and Appendix.

Supplementary Table

Study protocol.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Andrews, N., Stowe, J., Kirsebom, F. et al. Effectiveness of COVID-19 booster vaccines against COVID-19-related symptoms, hospitalization and death in England. Nat Med 28 , 831–837 (2022). https://doi.org/10.1038/s41591-022-01699-1

Download citation

Received : 10 December 2021

Accepted : 14 January 2022

Published : 14 January 2022

Issue Date : April 2022

DOI : https://doi.org/10.1038/s41591-022-01699-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Results of the cologne corona surveillance (cocos) project– a cross-sectional study: survey data on risk factors of sars-cov-2 infection, and moderate-to-severe course in primarily immunized adults.

• Max Oberste
• Teodora Asenova
• Martin Hellmich

BMC Public Health (2024)

Bivalent mRNA vaccine effectiveness against COVID-19 among older adults in Japan: a test-negative study from the VENUS study

• Yudai Tamada
• Kenji Takeuchi
• Haruhisa Fukuda

BMC Infectious Diseases (2024)

SARS-CoV-2 vaccination may mitigate dysregulation of IL-1/IL-18 and gastrointestinal symptoms of the post-COVID-19 condition

• Claudia Fischer
• Edith Willscher
• Christoph Schultheiß

npj Vaccines (2024)

Coffee as a dietary strategy to prevent SARS-CoV-2 infection

• Chen-Shiou Wu
• Yi-Chuan Li
• Mien-Chie Hung

Cell & Bioscience (2023)

The global COVID-19 vaccine surplus: tackling expiring stockpiles

• Nguyen Khoi Quan
• Nguyen Le My Anh
• Andrew W. Taylor-Robinson

Infectious Diseases of Poverty (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Effectiveness of Autumn 2023 COVID-19 vaccination and residual protection of prior doses against hospitalisation in England, estimated using a test-negative case-control study

Affiliations.

• 1 UK Health Security Agency, London, United Kingdom. Electronic address: [email protected].
• 2 UK Health Security Agency, London, United Kingdom.
• 3 UK Health Security Agency, London, United Kingdom; NIHR Health Protection Research Unit in Vaccines and Immunisation, London School of Hygiene and Tropical Medicine, London, United Kingdom; NIHR Health Protection Research Unit in Respiratory Infections, Imperial College London, United Kingdom.
• 4 UK Health Security Agency, London, United Kingdom; NIHR Health Protection Research Unit in Vaccines and Immunisation, London School of Hygiene and Tropical Medicine, London, United Kingdom.
• PMID: 38719110
• DOI: 10.1016/j.jinf.2024.106177

Introduction: The last COVID-19 vaccine offered to all adults in England became available from November 2021. The most recent booster programme commenced in September 2023. Bivalent BA.4-5 or monovalent XBB.1.5 boosters were given. During the study period, the JN.1 variant became dominant in England.

Methods: Vaccine effectiveness against hospitalisation was estimated throughout using the test-negative case-control study design where positive PCR tests from hospitalised individuals are cases and comparable negative PCR tests are controls. Multivariable logistic regression was used to assess vaccine effectiveness against hospitalisation with the test result as the outcome, vaccination status as the primary exposure variable of interest and confounder adjustment.

Results: There was no evidence of residual protection for boosters given as part of previous campaigns. There were 28,916 eligible tests included to estimate the effectiveness of the autumn 2023 boosters in those aged 65 years and older. VE peaked at 50.6% (95% CI: 44.2-56.3%) after 2-4 weeks, followed by waning to 13.6% (95% CI: -11.7-33.2%). Estimates were generally higher for the XBB.1.5 booster than the BA.4-5 booster, but this difference was not statistically significant. Point estimates were highest against XBB sub-lineages. Effectiveness was lower against both JN.1 and EG.5.1 variants with confidence intervals non-overlapping with the effectiveness of the XBB sub-lineages at 2-4 weeks for EG.5.1 where VE was 44.5% (95% CI: 20.2- 61.4%) and at 5-9 weeks for JN.1 where VE was 26.4% (95%CI: -3.4-47.6%).

Conclusions: The recent monovalent XBB.1.5 and bivalent BA.4-5 boosters provided comparable and good protection against hospitalisation, however there was evidence of lower VE against hospitalisation of these boosters against JN.1.

Keywords: COVID-19; immunisation; observational study; test-negative case-control; vaccine effectiveness.

Crown Copyright © 2024. Published by Elsevier Ltd. All rights reserved.