Sex difference in the discordance between Abbott Architect and EuroImmun serological assays

Introduction

In response to the emergence of sudden acute respiratory syndrome coronavirus 2 (SARS-CoV-2), the urgent need for assessment of population prevalence led to the release of serological assays with abbreviated validation under the Food and Drug Association’s Emergency Use Authorization (EUA) (Ainsworth et al., 2020; Karpe et al., 2018; Rikhtegaran Tehrani et al., 2020). While the initial results demonstrated high sensitivity and specificity, studies have found evidence suggestive that the clinical performance for some is much lower than initially anticipated (Prince et al., 2020; Tang et al., 2020). As the Abbott Architect SARS-CoV-2 immunoglobulin G (IgG) (Abbott) and EUROIMMUN SARS-COV-2 enzyme-linked immunoassay (ELISA) (EI) have often been treated as a comparable diagnostic in seroprevalence surveys, understanding their respective limitations is critical to sero-epidemiological investigations (Prince et al., 2020; Tang et al., 2020).

The Utah Health and Economic Recovery (HERO) Project is a seroprevalence survey aimed to improve the understanding of community-based SARS-CoV-2 activity. As a part of HERO, a sample of Utah residents took part in diagnostic screening and completed a survey assessing demographic and epidemiologic factors. Serum from most participants were dually evaluated by both Abbott and EI at the same encounter, thus the data collected throughout this project provides a unique opportunity to investigate their real-world performance and the factors contributing to underlying discordance.

While many studies have described sex-dependent differences in immune response and severity of coronavirus disease 2019 (COVID-19), no studies have previously described sex-associated discordance between immunoassays (Agrawal et al., 2021; Bunders & Altfeld, 2020; Grzelak et al., 2021; Haitao et al., 2020; Lakbar et al., 2020; Maleki Dana et al., 2020; Muecksch et al., 2021; Takahashi et al., 2020). This study evaluated discordance between Abbott Architect SARS-CoV-2 IgG and EUROIMMUN SARS-COV-2 ELISA in a real-world setting.

Materials & Methods

From June 10 to August 15, 2020, a total of 8,246 specimens were evaluated by both the Abbott Architect SARS-CoV-2 IgG (Abbott) and the EUROIMMUN SARS-CoV-2 ELISA (EI) assays as part of the HERO Project. The Abbott assay detects anti-SARS-CoV-2 IgG against the nucleocapsid protein (anti-N), and the EI assay detects anti-SARS-CoV-2 IgG directed against the S1 domain of viral spike protein (anti-S1) (Rychert et al., 2021; Tang et al., 2020). Of those screened by both diagnostic tests, 99% (n = 8,166) completed the survey assessing demographic and epidemiologic factors.

The University of Utah Institutional Review Board determined the HERO project to be non-research public health surveillance and waived the requirement for documented consent. The review board determined that use of these data for analysis to understand the dynamics of SARS-CoV-2 was exempt from further review (IRB_00132598).

Summary statistics stratified by Abbott and EI assay results were calculated for sex, age, race, ethnicity, diabetes, cardiovascular disease, hypertension, immunosuppressive therapy, autoimmune disease, self-reported previous positive polymerase chain reaction (PCR), positive PCR at the time of screening, known exposure to COVID-19 case, and history of anosmia or ageusia. Fisher’s exact tests were performed to evaluate significant association between assay results and demographic and epidemiological factors.

Sex-stratified phi correlation coefficients were calculated to quantify the degree of concordance between Abbott and EI assay’s quantitative results for the total sample population. Concordance was evaluated at the manufacturer-recommended and adjusted cut-points for Abbott (quantitative threshold: 1.0 to 1.8) and EI (quantitative threshold: 0.7−2.0). The manufacturer recommended cut-points for positivity were defined as 1.1 for EI and 1.4 for Abbott.

Participants who self-reported a previous positive PCR, had a positive PCR at the time of screening, claimed anosmia or ageusia after March 1, 2020, or claimed an exposure with a known case were classified as the high prevalence subgroup. These factors were utilized as they are potentially predictive of true positivity. The remaining participants were categorized in the low prevalence subgroup. While we did not know which of the discordant results was a true positive or false positive, by stratifying the results into these two subgroups we generated a group with a greater proportion of samples of false positives. This approach enabled the assumption that the positive predictive value was lower in the low prevalence subgroup; therefore, a greater proportion of people who tested positive were false positives. Sex- and subgroup-stratified summary statistics for the low prevalence subgroup were calculated and reported below. Differences between stratum were evaluated utilizing Fisher’s exact test.

Multivariable mixed-effect logistic models were implemented to evaluate the association between assay positivity and sex in the low prevalence subgroup while controlling for age, race, ethnicity, diabetes, cardiovascular disease, hypertension, immunosuppressive therapy, and autoimmune disease (Akama-Garren & Li, 2021; Chen et al., 2021a; Fox et al., 2020; Mouliou & Gourgoulianis, 2021; Nishiga et al., 2020). This analysis was limited to the variables captured in a self-reported survey as a part of the HERO Project. These covariates were selected due to their association with severity of illness, immune response, or known differences in the prevalence of COVID-19 in Utah (Akama-Garren & Li, 2021; Chen et al., 2021a; Fox et al., 2020; Mouliou & Gourgoulianis, 2021; Nishiga et al., 2020). Household ID was included in the model as a random intercept to account for clustering. Authors considered the use of a fixed effect and isotonic regression for this analysis, but felt a mixed-effect model was better suited to account for variability within and across participants as multiple participants from the same household may be captured during the study period. Odds ratios and 95% confidence intervals were evaluated. The variance inflation factor was evaluated to assess for multicollinearity using the performance v0.10.1 package.

Sensitivity analyses were performed to evaluate the impact of alternate cut points that yielded the greatest reduction without reducing the positive concordance in the low prevalence subgroups.

Statistical analyses were performed with R v4.0.3 (R Core Team, 2020). The mixed effect logistic regression was implemented with the lme4 v1.1-27 package (https://cran.r-project.org/web/packages/lme4/index.html).

Results

Discussion

This study evaluated the side-by-side performance of Abbott and EI in a seroprevalence survey. A significant qualitative discordance was observed between Abbott and EI. While several factors were identified to be strongly associated with this discordance, a significant sex imbalance was identified among EI positives that was not present among Abbott positives. Discordance between the evaluated assays is well described in the literature; however, the identification of sex-driven false positivity of EI relative to Abbott provides potential understanding of factors contributing to the described discordance.

As the focus of this investigation was to evaluate the issue of false positivity, we capitalized on the low prevalence subgroup, in which the proportion of false positives are likely to be higher, to evaluate factors contributing to discordant results. Overall, EI was associated with more false positives than Abbott, with a substantial imbalance between sexes. This association was isolated to EI and not evident with Abbott. While studies have identified a rapid decline in anti-N following infection among males (Grzelak et al., 2021), our choice to restrict our analyses to the low prevalence subgroup reduced the potential of this decline to impact the results. Further, as sex was not significantly associated with an EI-positive result in the high prevalence subgroup, our findings suggest a reduced positive predictive value for the EI assay in low prevalence populations.

Given the sex-centric discordance observed between assays, alternative sex-specific thresholds for a positive result were evaluated to determine the implications on false positivity. At the Abbott and EI recommended thresholds, females were associated with greater concordance than males. While the observed discordance among males was greater than females at all observed thresholds, a significant improvement in sex-specific concordance as the EI threshold was increased was observed with a minimal reduction in the frequency of positive concordant results. Sensitivity analyses of alternate threshold for EI positivity, (cut point: 1.4 and 1.8), produced similar results, likely due to a reduction in both male and female discordance. Reductions below the EI recommended cut-point for positivity were associated with significant reductions in concordance. These results are consistent with those in previous literature, which described that the classification of indeterminate results (quantitative value of 0.08−1.0) as positive negatively impacts the assay’s specificity (Prince et al., 2020; Stocking et al., 2022; Tang et al., 2020). These results suggest the need to evaluate the benefit of sex-specific positivity threshold to improve the performance of EI.

Seroprevalence studies are an essential tool for public health responses against infectious diseases and provide information for estimating transmission intensity and population susceptibility (Bryant et al., 2020). In the COVID-19 pandemic alone, thousands of seroprevalence studies have been conducted, with many used to provide data on the burden of infection in their respective regions (Chen et al., 2021b). An accurate seroprevalence estimate includes a correction for test performance, which is almost exclusively reported in a sex-independent manner. Considering these findings, it is important to evaluate sex-specific corrections for test performance in seroprevalence estimates.

Our results are consistent with the literature, which has described a variety of specificity limitations with EI (Prince et al., 2020; Tang et al., 2020). This includes EI’s Information for Use which suggests potential cross-reactivity with several commonly circulating respiratory pathogens, including respiratory syncytial virus. While we were unable to control for cross-reactivity in this analysis, Nawrocki et al. (2021) identified a dramatic decline in several of these pathogens early in the pandemic due to non-pharmaceutical interventions aimed at reducing SARS-CoV-2. This nationwide reduction in pathogens of potential cross-reactivity during the study period and the magnitude of the discordance suggest the observed association is unlikely to be due to cross-reactivity alone. However, this impact should be reevaluated as common respiratory pathogens return to pre-pandemic levels.

This evaluation was limited to the data collected as a part of the HERO Project in Utah. Due to the racial homogeneity of the population, further investigations are necessary to evaluate if the findings hold for additional races.

Lastly, investigators had limited resolution into true positivity or time since positivity, limiting the ability to assess sensitivity in the low prevalence subgroup. Further evaluation of sex-specific sensitivity and specificity of anti-N and anti-S assays is necessary to provide insight into the potential limitations and benefits of sex-specific thresholds for EI and evaluate the generalizability to other anti-S assays. Further, a better understanding of the effect of sex on the performance of these assays is critical in the post-vaccination era to effectively measure the population’s immune response to the vaccine and previous infections.

Conclusions

Significant quantitative and qualitative discordance was observed between Abbott and EI throughout the seroprevalence study. Higher positivity was observed in EI than among Abbot. Investigators observed disproportionately positive EI results among men; however, this was not observed with Abbott. The association between the male sex and EI seropositivity was particularly strong in the low-prevalence subgroup. Our results suggest there may be significant sex-associated specificity limitations with the EI assay. As these findings may extend to other anti-S assays utilized for SARS-CoV-2 seroprevalence investigations, further investigation is needed to evaluate the generalizability of these findings with other manufactures’ assays.

Supplemental Information