Heterogeneity of the effect of the COVID-19 pandemic on the incidence of Metabolic Syndrome onset at a Japanese campus

Study design

This was an observational study that utilized previously collected data.

Data source and participants

This study included data obtained from workers at the Shikata Campus of Okayama University. As this campus has various departments, comparisons were performed among the departments. Although particular attention was given to medical workers, data from all departments were used for improved elucidation of the characteristics of the high-risk group and the accuracy of the causal inference model. The dataset was provided by the Junpukai healthcare center, which conducts health checkups at the campus.

All participants met the following inclusion criterion: workers who underwent health checkups for at least two consecutive years from 2017 to 2020. The following individuals were excluded from the study: (1) individuals who only had data unrelated to general health assessments, (2) individuals who were not tested for MetS, and (3) individuals with MetS at baseline. Health checkup data obtained in 2017, 2018, and 2019 were the baseline data with the respective follow-ups in 2018, 2019, and 2020. All data that were obtained for this study met the selection criteria.

As this study only used pre-existing information, an a priori sample size calculation was performed. To ensure the reproducibility of the analysis, the dataset and analysis code were made publicly available. However, to protect personal information, personal identifiers were removed; further, quasi-identifiers were anonymized by resampling and randomization. Anonymized data were used for analysis in this study. However, subtle discrepancies emerged between the original and anonymized datasets during the anonymization process. Although privacy concerns preclude the disclosure of the original dataset, a comparison was conducted to ascertain any differences in the analytical results between the two datasets.

Outcome variable

Follow-up data were used to determine whether participants had MetS based on the diagnostic criteria provided by the Ministry of Health, Labour and Welfare (MHLW) (short URL: https://bit.ly/3Jx9b8g).

Based on Japan’s MetS diagnostic criteria (Journal of the Japanese Society of Internal Medicine, 2005), a flowchart was crafted from health checkup data of workers, considering challenges such as missing data and non-fasting blood glucose levels. It aims to provide accurate diagnoses under these constraints, hence its complexity. Differences from international criteria (Alberti et al., 2009) are detailed in Article S1, with the English flowchart.

Both Japanese and international criteria use similar tests, but with different cutoffs: high-density lipoprotein (HDL) cholesterol (40 mg/dL), triglycerides (150 mg/dL), fasting blood sugar (110 mg/dL), and blood pressure (130/85 mmHg) for Japan. The international criteria have sex-specific HDL thresholds, whereas Japan has uniform thresholds. Additionally, Japan combines triglyceride and HDL cholesterol as one item, unlike the international approach. Japan’s cutoff for blood glucose differs from the international standard (100 mg/dL), and they use different abdominal circumference measurement points. Internationally, it is midway between the lower rib margin and the iliac crest; in Japan, it is at the umbilicus level. This alters cutoffs: internationally, 90 cm (males) and 80 cm (females); in Japan, 85 cm (males) and 90 cm (females). Japan defines MetS as having excess visceral fat, indicated by abdominal circumference, plus two or more cardiovascular risk factors (hyperglycemia, dyslipidemia, high blood pressure).

Exposure variable

The exposure was defined as the COVID-19 pandemic. As the COVID-19 pandemic in Japan began in January 2020, workers who underwent follow-up health checkups in 2018–2019 and 2020 were considered as “unexposed” and “exposed”, respectively. The definition of exposure status is more clearly shown in Fig. 1.

Predictive variables

Health-checkup items, sex, age, and working department obtained during baseline health checkups were used as predictor variables. Health-checkup items are summarized in Article S2. Missing continuous variables were imputed using median values; indicator variables were used to indicate missing data. Missing categorical variables were assigned to a separate missing category. Departments were divided into four categories: administrative (library, cafeteria, etc.), research, medical (excluding the intensive care department), and intensive care department.

Statistical analysis

Descriptive statistics were used to analyze baseline health checkup data. Continuous variables are presented as means with standard deviations, while categorical variables are presented as frequencies and percentages. Missing values were excluded during the computation of descriptive statistics.

To assess the impact of the COVID-19 pandemic on the incidence of new-onset MetS, the conditional average treatment effect (CATE) was calculated. CATE is defined as a difference between the expected outcomes of two treatments based on covariates, representing an individualized causal effect in causal inference (Abrevaya, Hsu & Lieli, 2015; Jacob, 2021; Kato & Imaizumi, 2023). CATE for the covariate X = x is defined using the flowing formula (Nie & Wager, 2021). Note that Y¹ and Y⁰ are the potential outcomes of exposure and non-exposure, respectively. ${\displaystyle \mathrm{CATE}\left(x\right)=E\left(Y^1-Y^0|X=x\right).}$

The main benefit of using CATE is its ability to uncover heterogeneity in treatment effects across subpopulations, which is crucial for personalized treatments and decision-making in various domains such as economics and healthcare (Jacob, 2021). In contrast to ATE, which estimates an average treatment effect for the entire population, CATE provides a more detailed understanding by estimating the effect specifically for subgroups within the population, thereby addressing the heterogeneity that ATE may overlook. For example, in economics, research with the advantages of CATE has been conducted (Crépon et al., 2015).

Traditionally, stratified analyses and analyses incorporating interaction terms have been employed to calculate the CATE. Contrastingly, this study utilized X-learner, which is a machine learning-based causal inference method that allows more flexible CATE calculation (Künzel et al., 2019a; Künzel et al., 2019b; Jacob, 2021). Furthermore, 95% confidence intervals (95% CIs) for CATE were computed using the bootstrap method (15,000 iterations) (Künzel et al., 2019b).

Values of CATE were calculated for the overall population, department, sex, and age. The difference in CATE was calculated for each category (division and sex). For division, I calculated the difference between the medical division and the other three divisions, using the medical division as a control. For sex, I calculated the difference between the females and the males, using the female as a control. Scatter plots were generated to illustrate the association between the CATE and age. Approximate curves with locally weighted regression smoothing were applied to the scatterplot to demonstrate the heterogeneity of the association between CATE and age according to the department. The population characteristics with CATE values in the upper and lower 10th percentiles were compared, and the standardized differences were calculated. All statistical analyses were performed using Stata 17/MP8 (StataCorp LLC, College Station, TX, USA). Statistical significance was set at p < 0.05.

Machine learning model: X-learner

Figure 2 presents a schematic representation of the X-learner method. Figure 2 was created based on the website (Matheus, 2023). First, models were constructed to predict outcomes using predictive variables in the exposed and non-exposed subgroups. The logistic regression model was used as the base learners M0 and M1, as the outcome was binary, and the computational load was very low. The area under the curve (AUC) was used to assess predictive utility; subsequently, the set of predictive variables with the highest AUC values in the 10-fold cross-validation was selected. Sets of predictive variables were selected from the original variables, squared variables, and interaction terms (Article S3).

Next, the imputed treatment effect (ITE) was calculated using the difference between the predicted and observed outcomes for the exposed and unexposed subgroups. Accordingly, there were no unexposed and exposed ITEs in the exposed and unexposed groups, respectively.

Subsequently, models were constructed to predict the ITE using predictive variables in the exposed and non-exposed subgroups (base learners: M2 and M3 in Fig. 2). The response variable in this prediction model, ITE, was assigned a real number in the range from −1 (100% prevention of new-onset MetS after exposure) to 1 (100% occurrence of new-onset MetS after exposure). As there are no appropriate regression models for this ITE, a beta regression model was used after transforming the data by adding 1 to the original value and dividing by 2 (new range: 0–1). The root mean squared error (RMSE) was used to evaluate the predictive utility; additionally, the set of predictive variables with the lowest value in the 10-fold cross-validation was selected (Article S3).

Finally, the weighted average of the ITE predictions (CATE1 and CATE0 in Fig. 2) obtained from M2 and M3 was used to estimate the CATE. Conditional exposure probabilities (i.e., propensity scores), which reduce bias through simulations (Künzel et al., 2019a; Künzel et al., 2019b), are usually used for weighting. However, as the exposure variable in this study was independent of covariates, i.e., health checkup results did not influence whether an individual experienced the COVID-19 pandemic, marginal exposure probability was applied for weighing.

This study used the following three assumptions to apply the X-learner approach for causal inference. First, it is assumed that enough covariates have been measured to calculate potential outcomes from these covariates accurately. Second, it is postulated that potential outcomes remain consistent before and after the COVID-19 pandemic, given the covariates. Furthermore, in calculating the ITE from the covariates, there is an assumption that an adequate number of covariates are measured to ensure precise calculation of ITE.

Ethical approval

This study was ethically approved by the clinical research review committee of the Junpukai healthcare center (approval date: June 3, 2022, approval number: 20210015). As this study only used pre-existing information and the data had been anonymized, informed consent had already been obtained through information disclosure and opt-out provisions negating the need to obtain consent from each individual.

Descriptive statistics

The dataset comprised data obtained from 13,993 health checkups. After excluding irrelevant data, data with missing outcomes, and data showing MetS at baseline, the total number of records was 3,572 (non-exposed, n = 2,181; exposed, n = 1,391). Subsequently, the data were anonymized and analyzed (Fig. 3).

In the original data, 629 individuals had only one measurement, 612 had two, and 2,331 had three; however, the same person could not be identified in the anonymized data used for analysis, which should generally be avoided with respect to independent and identically distributed variables. Despite this situation, the overall impact of the violation of independence assumption is mitigated by several factors. Firstly, no individual had repeated measures in the exposed data subset, thus upholding the independence assumption in estimating the models m1 and m3 depicted in Fig. 2. Conversely, some individuals had up to two measurements in the unexposed data subset, suggesting a potential breach of the independence assumption. However, the effect of this violation is minimal. This is primarily because the anonymization process involved adding random numbers to the variables, reducing the correlation within individuals’ repeated measurements. Thus, although the original data might have breached the independence assumption, the analyzed anonymized data largely preserves this principle. Additionally, the main objective of the analysis was not to derive regression coefficients but to develop outcome prediction models (as shown in m0 and m2 of Fig. 2). These models were evaluated using 10-fold cross-validation AUC and RMSE metrics, demonstrating non-poor performance. Furthermore, the sensitivity analysis also confirmed that the effect of violating the independence assumption was not substantial (Article S3). Therefore, any potential violation of the independence assumption has a limited effect.

Table 1 presents the descriptive statistics results. Most participants were females (1,544 (70.8%) and 866 (62.3%) in the unexposed and exposed groups, respectively). The mean age (±standard deviation) was 48.2 ± 8.2 and 47.8 ± 8.3 years in the unexposed and exposed groups, respectively. There was a slight difference in the incidence of new-onset MetS during the follow-up period between the unexposed (n = 66 (3.0%)) and exposed (n = 72 (5.2%)) groups, respectively.

There was no significant between-group difference in the baseline urinalysis and blood test results (Table 2).

Evaluation of base learners

According to the logistic regression model, the AUC values for M0 and M1 were 0.95055 and 0.89743, respectively. According to the beta regression model, the RMSE values for M2 and M3 were 0.10096 and 0.14933, respectively (Article S3).

Model M0 exhibited outstanding discrimination with an AUC exceeding 0.9, while model M1 demonstrated excellent discrimination, having an AUC above 0.8, in line with Hosmer and Lemeshow’s criteria (Hosmer Jr, Lemeshow & Sturdivant, 2013).

For models M2 and M3, RMSE values ranged from 0 (complete accuracy) to 1 (complete inaccuracy), reflecting real-number outcomes between 0 and 1 in a beta regression model. The RMSE results revealed an average error of approximately 10–15%. Despite the absence of a standard benchmark for RMSE interpretation, these values were deemed acceptable, not significantly undermining the study’s validity, and thus, the research was continued.

CATE estimates

As shown in Table 3, the CATE estimate for the overall population (n = 3,572) was 0.044 (95% CI [0.008–0.080]; p = 0.017). This indicates that the COVID-19 pandemic increased the probability of new-onset MetS by 4.4%. Figure 4 shows a histogram depicting this distribution, with CATEs being widely distributed but most frequently occurring around 0 (i.e., no causal effect). However, 389 (10.4%) participants had a CATE > 0.25.

According to the department, the CATE values in the administrative, research, medical, and intensive care departments were 0.034 (95% CI [−0.025–0.094]; p = 0.256), 0.068 (95% CI [−0.016–0.153]; p = 0.112), 0.032 (95% CI [−0.012–0.077]; p = 0.158), and 0.154 (95% CI [−0.021–0.329]; p = 0.084), respectively. This indicated heterogeneity in the impact, with the intensive care department having a particularly high CATE.

Regarding sex, the CATE values for female and male workers were 0.033 (95% CI [0.005–0.062]; p = 0.022) and 0.066 (95% CI [−0.016–0.148]; p = 0.116), respectively, indicating heterogeneity between sexes.

Table 4 shows the CATE differences between each category (department and sex). For the department, the medical division was used as the reference category. The CATE differences in the administrative, research, and intensive care departments were 0.002 (95% CI [−0.070–0.074]; p = 0.951), 0.036 (95% CI [−0.054–0.127]; p = 0.434), and 0.122 (95% CI [−0.054–0.298]; p = 0.173), respectively. For sex, males had a slightly higher CATE than females at 0.032 (95% CI [−0.049–0.114]; p = 0.435). However, none of the CATE differences were statistically significant.

Figure 5 shows a scatter plot and smoothing curves depicting the relationship among CATE, age, and department. Although the plot points are widely distributed, the smoothing curves revealed a gradual increase in CATE with age. Furthermore, the intensive care department showed a higher CATE than the other departments at all ages.

Characteristics of the high-CATE group

Tables 5 and 6 present the characteristics of the populations with the upper 10th percentile of CATE values (high) and the lower 10th percentile (low). The high group had a CATE of 0.261 or higher, and the low group had a CATE of −0.157 or lower.

The difference between the high- and low-CATE groups was especially large for the following items (standardized differences): age (0.766), history of treatment for hyperlipidemia (1.099), urinary protein (0.585), alanine aminotransferase levels (0.500), γ-glutamic pyruvic transaminase levels (0.611), fasting triglyceride levels (0.600), and hemoglobin A1c (0.501).

Results based on the original dataset

There were no significant differences in the results after analysis of the original and anonymized datasets. CATE values calculated using the original dataset are presented in Tables S1 and S2 and Figs. S1 and S2.

Summary of the results

The COVID-19 pandemic increased the probability of new-onset MetS by 4.4%. This impact had heterogeneity; moreover, there was a larger impact in the intensive care department (15.4% increase) than in other departments. The high-CATE group was characterized by older age, urinary protein, elevated liver enzymes, higher triglyceride levels, increased hemoglobin A1c levels, and a history of treatment for hyperlipidemia.

Interpretation of the results

In this study, the CATE differed according to age and sex. Older age was associated with higher CATE values; however, there was substantial interindividual variability. The prevalence of MetS is typically higher among elderly individuals (Arai et al., 2006; Hirode & Wong, 2020), which may explain their increased vulnerability to new-onset MetS during the pandemic. Moreover, the pre-pandemic prevalence of MetS was higher among males than among females (Arai et al., 2006; Kudo et al., 2021).

Furthermore, this study observed heterogeneity according to the department. The lower CATE in the healthcare department (excluding the intensive care department) may be attributed to several factors. For example, as healthcare workers are relatively more knowledgeable about health, they may have continued engaging in activities (e.g., exercise and diet) to maintain their health during the COVID-19 pandemic. Additionally, some reports have suggested favorable behavioral changes among employed and highly educated individuals (Knell et al., 2020), which may explain some of the behavioral changes among healthcare workers.

Conversely, the CATE was higher among intensive care workers, which could be attributed to long working hours as well as high physical and psychological loads, including handling patients with COVID-19 (Gupta & Sahoo, 2020; Koontalay et al., 2021).

The high-CATE group was characterized by older age, urinary protein, elevated liver enzymes, higher triglyceride levels, increased hemoglobin A1c levels, and a history of hyperlipidemia treatment, which are indicative of suboptimal healthcare before the COVID-19 pandemic. Such individuals are more susceptible to new environmental stressors and may be at an increased risk of developing new-onset MetS.

Additionally, the marked difference in CATE values between sexes underscores the need for sex-specific prevention strategies, particularly in high-stress environments such as intensive care units. This indicates a nuanced interplay of biological and environmental factors contributing to MetS risk, which warrants further investigations to tailor preventive strategies effectively.

Comparison with previous studies

This study showed both similarities and differences in the characteristics of workers whose health conditions are likely to deteriorate compared with previous studies on the adverse health effects of the COVID-19 pandemic on frontline healthcare workers, especially with respect to mental health (Batra et al., 2020; Gupta & Sahoo, 2020; De Kock et al., 2021; Koontalay et al., 2021). A previous study reported that female workers, workers with fewer enrollment years, and intensive care workers were at an increased risk of mental illness (Azoulay et al., 2020; Matsuo et al., 2020; Zhu et al., 2020; Marvaldi et al., 2021; Smallwood et al., 2021). Considering the results of previous and present studies, intensive care workers are observed to be more susceptible to both mental health issues and MetS. Notably, a sex-based disparity exists wherein females predominantly face mental health challenges, while males are more prone to MetS. Age also delineates this distinction; workers with fewer enrollment years (i.e., many younger workers) tend to exhibit mental health concerns, whereas older individuals are more likely to develop MetS.

A history of mental health disorders has been found to predispose individuals to anxiety and depressive symptoms during the COVID-19 pandemic (Zhu et al., 2020; Arpacioglu, Gurler & Cakiroglu, 2021; Smallwood et al., 2021). In this study, the high-CATE group was characterized by a history of hyperlipidemia treatment and high laboratory values. Although the previous and present studies differ in diseases, both show that a history of pre-pandemic illness is associated with a poor health status during a major event such as a pandemic.

Suggestions for prevention interventions

The present findings indicate the need for preventive interventions for the described high-CATE population during a pandemic. For example, it might be beneficial to inform this population regarding their health vulnerability during a pandemic. Moreover, strict treatment management should be thoroughly implemented for individuals with pre-existing conditions. During a pandemic, home exercises (Chtourou et al., 2020) should be encouraged and assistance provided to create an environment conducive for home exercises. Pappa, Sakkas & Sakka (2022) suggested the importance of enhancing social support networks to promote psychological resilience. Several interventions can be implemented to reduce social isolation during a pandemic, including psychotherapy, friendship lessons, robot pets, social facilitation, coping strategies, and further education on the causative disease (Williams et al., 2021; Gonçalves et al., 2022). Another viable intervention for mitigating the isolation requirements of the COVID-19 pandemic is videoconferencing (Williams et al., 2021).

Strengths and limitations of the study

This study has several strengths. First, this is the first study on the incidence of new-onset MetS among healthcare workers during the COVID-19 pandemic. Second, as this study used health checkup data, there were few misclassified data items. Third, the large sample size (n = 3,572) considerably reduced the influence of random errors. Fourth, the X-learner model allowed the flexible calculation of CATE, which captured the characteristics of the high-risk population that could not be identified by ATE alone.

Nevertheless, this study has some limitations. First, this was a single-center study, and generalizability of the study findings to other universities and hospitals cannot be assumed. Second, this study did not consider some predictor variables, including job type (doctors, nurses, and so on), working hours, stress levels, socioeconomic status (income and education level), physical activity, and social support. Incorporating these variables would allow the construction of a more accurate prediction model of base learners, which leads to less bias. However, some measured variables can serve as surrogate variables for unmeasured variables. By including surrogate variables in the statistical model, the impact of unmeasured variables will be minimized to some extent. For example, body mass index at baseline could be a surrogate variable for physical activity. Third, the exclusion of health checkup data of patients without MetS (4,098 unexposed, 1,892 exposed) could raise concerns regarding selection bias. However, the absence of MetS data was due to participants’ younger age, as according to Japanese law, some MetS testing may be omitted for young workers. Thus, these exclusions are independent of exposure, and the resulting bias is expected to be minimal. Fourth, base learners M2 and M3 had limited performance. Instead of the beta regression model used in this study, a random forest regression model may allow an improved prediction accuracy. However, as the response variable in this study was a real number ranging from −1 to +1, random forest regression may also predict values outside this range. Furthermore, the same person was included in both the exposed and unexposed groups, which should generally be avoided with respect to independent and identically distributed variables. However, as indicated in the second paragraph of the Results section, the impact of this violation is considered limited. Finally, calculating the CI using the bootstrap method has been shown to result in narrow CIs for the X-learner estimates (Künzel et al., 2019b); therefore, the calculated CIs may not have 95% coverage.