Comparison design and evaluation power in cohort and self-controlled case series designs for post-authorization vaccine safety studies

Cohort study design

Vaccine safety is evaluated by identifying a target population, stratifying it based on receipt of the vaccine of interest, and comparing the incidence proportion and rate of specific AEFIs within each group. Crucially, adjusting for confounding factors is necessary to appropriately estimate a effect. In addition to confounding, we also need to be aware of selection bias due to discrepancies between target population and analysis population, loss-to-follow-up; and measurement error due to measurement of variables (Lash et al., 2021). In particular, we need to pay attention to the measurement bias of data from claims data (Yamana et al., 2017; Newcomer et al., 2018; Hempenius et al., 2021). This design enables the estimation of relative (ratio) effect measures and absolute (difference) effect measures. The assessment of rare AEFIs necessitates large cohorts.

SCCS design

SCCS design is a case-only design. The analysis population comprises individuals who have experienced an AEFI of interest. The observation period for individuals is segmented into a risk period and a control period, and the period during which the AEFIs occurred is utilized to estimate the incidence rate ratio. The risk period is defined as the period during which the AEFIs was considered to be clinically related to the vaccination. By creating within-individual comparisons, time-invariant confounders could be automatically adjusted for. Conversely, SCCS designs also have time-varying confounding (e.g., age in pediatric vaccines). Like cohort studies, SCCS studies should be alert to selection and measurement biases arising from the data. In particular, the SCCS target of estimation, the incidence rate ratio, is susceptible to measurement error related to the time of outcome occurrence and exposure initiation.

Power analysis in simulation

The power to detect rare AEFIs in a PASS was calculated for each design. For the cohort design, we calculated power considering different risk ratios, prevalences of the unvaccinated, matched ratio of vaccinated to unvaccinated, and varied sample sizes of vaccinated individuals. In this simulation, individuals who were not vaccinated were paired with vaccinated individuals randomly to define the observation period for the non-vaccinated, rather than to control for confounders. This ratio of pairs is called the “matched ratio” in this article. We estimated power by estimating p-values using the Fisher’s exact test. Note that to assess the behavior of power in scenarios devoid of bias in the estimates, we established our setting to be free from confounding. In parallel, for the SCCS design, we calculated power considering different rate ratio, observed period, risk period, and event counts. We derived p-values via Wald statistics and used the SCCS package in R (Weldeselassie, Whitaker & Farrington, 2022). For both designs, the null hypothesis was that relative ratio = 1, with two-sided significance level of 5%. The detailed settings for each design are summarized in Table 1, following the recommendations of Morris, White & Crowther (2019). Assuming that vaccination increases the risk of AEFIs, only the positive direction was evaluated. The number of simulation iterations was set to 5,000 and implemented using R version 4.3.0 (R Core Team, 2023). The code is archived at GitHub: https://github.com/ShuntaroS/23-power-cohort_SCCS-PASS.

Cohort study design

Figure 2 illustrates the power trend. When the prevalence of the unvaccinated group was less than 1 in 100,000, even with a risk ratio of 2, a sample size of 100,000 vaccinated individuals did not demonstrate sufficient power to detect the effect. Assuming a prevalence of approximately 50/100,000, a near-null risk ratio may be detectable within a sample size of about 100,000. When the matched ratio of vaccinated to unvaccinated was altered to 1:1, 3:1, or 5:1, efficiency improved with a change in the ratio from 1:1 to 3:1. However, modifying the ratio from 3:1 to 5:1 did not increase efficiency.

SCCS design

Figure 3 shows the trend of detection power for a risk period of 42 days. The power trend for a risk period of 14 days is shown in Fig. S1, and the power trend for a risk period of 30 days is shown in Fig. S2. The “saw-tooth phenomenon” was particularly pronounced when the observation period was longer than 3 years. The power did not increase with the expanded observation period, but rather decreased. Furthermore, when we observed the trend in power by risk period, power increased as the ratio of the risk period to the observation period became larger. When the incidence ratio was small, a substantial quantity of event cases was necessary.

Cohort study design

When the effect size is small and the non-vaccinated group’s prevalence is low, detecting any effect can be challenging with a realistically achievable sample size. In such cases, it is better to determine the sample size needed based on the clarity of the effect-size confidence interval before starting the study, rather than relying purely on hypothesis testing. If the calculated confidence interval falls within the pre-specified confidence interval, we may infer that there is likely no association between vaccination and the AEFIs under consideration (Lash et al., 2021). A Bayesian method could be considered if the statistical significance is not a concern in the frequentist statistical approach (Greenland et al., 2016).

In a cohort study design, evaluating the association between vaccination and an AEFIs involves several steps: handling data, matching of nonvaccinated individuals with vaccinated individuals for index date, and estimating with adjustments for confounders (using an outcome regression model or propensity score model (Hernán & Robins, 2020)). Since data handling and pairing demand significant computational resources, a ratio of three vaccinated individuals to each non-vaccinated individual is considered a good balance for efficiency.

Finally, we have consistently used the Fisher’s exact test for simulation of cohort studies, since the number of events may be extremely small in some cases. However, when the number of events is large, simulations based on the normal approximation should also be considered. The normal approximation method has better power than the Fisher’s exact test.

SCCS design

It was anticipated that a longer observation period would lead to a greater number of events, thereby increasing the power. However, findings revealed that the power actually increased with shorter observation periods. As such, the observation period should be set with consideration for the number of events that can be collected within that timeframe. A preliminary estimate of the number of events per observation period could assist in study design.

Focusing on the risk period, the power increased as the length of the risk period relative to the observation period increased. Since the length of the risk period is related to the AEFI setting, it is not desirable to easily vary the risk period. Thus, the ratio of the risk period to the observation period should also be taken into account in setting the observation period.

As the observation period lengthened, the instability of the power trend became more pronounced. This instability is referred to as the “saw-tooth phenomenon”. This phenomenon occurs in power calculations involving discrete data and was observed in SCCS analyses due to the discrete nature of the event counts both inside and outside the risk period (Farrington, Whitaker & Weldeselassie, 2018). Based on these results, we recommend that the power calculation not be performed only for a specific observation period and sample size; instead, both the observation period and sample size be varied over a somewhat range to check the transition of power.

One assumption of SCCS design is that within an individual, events can be recurrent and independent; non-recurrent events are rare (Farrington, Whitaker & Weldeselassie, 2018). As observation periods shorten, the chance of observing the AEFIs of interest only once in a PASS increases. In this case, the rarity of the event becomes important. Whitaker, Steer & Farrington (2018) reported that a cumulative incidence rate of less than 10% during the observation period would indicate a small bias, suggesting that in a PASS, the AEFIs of interest can be considered “rare”. However, the power may be unstable when the number of events is small. Thus, even when applying the SCSS, a cohort comprising 10 million individuals is required for an AEFIs with an incidence exceeding 100 events, which would correspond to the incidence of one event in every 100,000 individuals.

Comparison of two designs

In a PASS, we want to know not only the association between vaccination and AEFIs, but also the association of each vaccine if possible. Thus, confounding should be reduced as much as possible. Cohort study designs necessitate adjustment for numerous confounders. When events are infrequent, an outcome regression model might not adequately account for confounding. A propensity score model could sufficiently adjust for these factors, providing a valid effect estimate. However, because we are matching non-vaccinated persons against vaccinated persons, it may be preferable to take time into account instead of the usual propensity score estimation (Andersen et al., 2016). By contrast, an SCCS study can theoretically eliminate confounding, suggesting a high degree of validity in the effect estimation. Moreover, it is less computationally demanding than cohort studies, as the analytic population can be limited to instances where an outcome is present. Despite these advantages, a PASS based on this design requires a large cohort to ensure robust power due to the rarity of events.

Regarding estimable association and effect measures, cohort studies may be more beneficial for public health because SCCS studies can only estimate relative measures (incidence rate ratios). By contrast, cohort studies can estimate absolute measures (risk differences). Absolute and relative measures are both required when deciding whether to endorse the safety of certain vaccines.

In this study, we have evaluated the power of both designs based on hypothesis tests, but since the association between vaccination and AEFIs may well be close to null, a precision-based design or a Bayesian approach may also be helpful.

Strengths and limitations

This study’s strength is grounded in the evaluation of research design and power within a realistic PASS setting. However, it also presents certain limitations. First, we did not use a model assuming confounding structure in our simulations because we focused our attention only on the power transition. Second, in SCCS studies, we observed a decrease in power as the observation period increased. Further theoretical study of this behavior is needed.