Development and validation of a novel risk score to predict 5-year mortality in patients with acute myocardial infarction in China: a retrospective study

Study populations and procedure

We conducted a retrospective observational study of 3,088 inpatients with International Classification of Disease (ICD) Version 9 diagnosed AMI based on the results of the coronary angiography in Hunan Province, China. Only patients with type 1 AMI, defined as plaque-mediated culprit lesions, were included in our study. The exclusion criteria were age younger than 18 years or older than 90 years; the length of hospital stay <1 day with a lack of information about the entire hospitalization for AMI, including information on death; severe hepatic impairment (alanine transaminase (ALT) > 400 U/L or total bilirubin (Tbil) > 340 µmol/L); malignancy; life-threatening infection; severe renal impairment (estimated glomerular filtration rate [eGFR] < 15 mL/min); severe and active autoimmune disease or haemodynamically unstable trauma at admission. This study was approved by the Ethics Committee of the Third Xiangya Hospital of Central South University and Xiangtan Central Hospital (IRB approval numbers: 2019-S489 and 2020-11-001, respectively).

For the development cohort, 1,723 patients with AMI who were admitted between January 2013 and December 2017 were consecutively recruited for this study from the Third Xiangya Hospital of Central South University in Changsha city. The last date of follow-up for the mortality was August 31, 2021. Patients with severe hepatic dysfunction (n = 58), severe renal impairment (n = 85), active malignancy (n = 54), severe autoimmune disease (n = 8), current life-threatening infection (n = 39), and haemodynamically unstable trauma (n = 4) at admission as well as those with a length of hospital stay <1 day (n = 4) were excluded. The remaining 1,471 patients were included in the final analysis of the development cohort. Among the patients in the final analysis, 292 patients died, and nine patients were censored during the follow-up period.

For the external validation cohort, 1,365 patients with ICD-9-confirmed AMI who were admitted between January 2013 and December 2017 were consecutively included from the Xiangtan Central Hospital in Xiangtan city. The last date of the follow-up was August 31, 2021. Patients with severe hepatic dysfunction (n = 1), severe renal impairment (n = 24), active malignancy (n = 16), autoimmune disease (n = 5), and haemodynamically unstable trauma (n = 2) at admission as well as those with a length of hospital stay <1 day (n = 66) were excluded. Thus, 1,251 patients in the external validation cohort were included in the final analysis. Among these patients, 186 patients died, and 65 patients were censored during the follow-up period.

The demographics and clinical characteristics, medical history, laboratory tests, imaging examinations, and in-hospital interventions were obtained from patients’ electronic medical records. Mortality and its associated causes were collected from the death registry reporting system of the National Center for Disease Control and Prevention (CDC) and were further confirmed through telephone follow-up. The clinical data for the construction of the predictive models were based on the measurements at admission for both cohorts. The ethics committee of each hospital approved the study protocols. The need to obtain informed consent from the patients was waived by both ethics committees. The study complied with the requirements in the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement (Table S1) (Moons et al., 2015). A flowchart of the model development and validation process is shown in Fig. 1.

Variable selection and model development

Forty-five candidate variables were identified from the clinical variables available at hospital admission and a literature review and entered into the selection process (Table S2). Of these, candidate variables with less than 20% missing data were included in the study (Table S3). The variables selected for the predictive risk model were chosen from a total of the forty-five candidate baseline parameters in a two-round process.

Least absolute shrinkage and selection operator (LASSO) regression was applied in the first-round of variable selection in the development cohort. LASSO regression is capable of minimizing the potential collinearity of variables and overfitting with the minimum lambda (λ = 0.037), and the mean squared error (MSE) is within one standard error of minimal MSE using ten cross-validations (Tibshirani, 1997; McClelland et al., 2015). Eighteen variables were selected from the first-round variable selection and entered into the second-round variable selection. To further validate the variable selection, we repeated the random forest algorithm 100 times in the development cohort. The variables were evaluated by the importance and the out-of-bag (OOB) value (Kruppa et al., 2014; Speiser, Durkalski & Lee, 2015).

The second round of variable selection was used to further select variables by multivariate stepwise backward regression model. Five iterative processes were applied to develop a parsimonious model while minimizing overfitting. An Akaike information criterion (AIC) value <4.6 and a p-value > 0.05 were the cut-off values for the chosen final variables. Ten variables were selected from the second round of variable selection.

In the final model building stage, ten variables selected from the second-round of variable selection were utilized in a multivariate Cox hazards regression model to develop an algorithm for predicting 5-year all-cause mortality in AMI patients. Due to the unknown individual absolute survival rate, we could obtain only the relative risks of AMI patients compared with those without any risk factors based on the regression coefficients of all variables (Kalbfleisch & Prentice, 2002; Royston & Altman, 2013). To improve the application of the C2ABS2-GLPK score in clinical practice, we transformed the Cox model into a nomogram model. Every selected variable was given a continuous score from 0 to 100 corresponding to the regression coefficient. The total risk score incorporating the ten predictors corresponded to the relative survival rate.

The X-tile program was used to identify the optimal cut-off value of the risk score to stratify patients with various degrees of mortality risk in the development cohort. The optimal cut-off values and patient groups were based on the minimal p-value (0.003) and the maximum of chi-square log-rank value (p-value < 0.001) according to 5-year mortality in patients with AMI. The optimal cut-off values were also validated in the external validation cohort (log-rank p-value < 0.001) (Camp, Dolled-Filhart & Rimm, 2004).

X-tile analysis revealed that patients with C2ABS2-GLPK scores ≤148 points were stratified into a low-risk group with a 5-year mortality of 6.6%; Patients with C2ABS2-GLPK scores between 149 and 218 points were graded as medium-risk with a 5-year mortality of 15.2%; and patients with C2ABS2-GLPK scores ≥219 points were categorized into a high-risk group with a 5-year mortality of 37.3%. Moreover, based on our literature review, the predicted mortality in each risk subgroup was similar to that of the GRACE score, thrombolysis in myocardial infarction (TIMI) risk score and modified GRACE score during a follow-up period of more than 5 years (Fox et al., 2010; Hall et al., 2018; Kozieradzka et al., 2011; Henderson et al., 2015).

Evaluation of model performance

The performance of the risk prediction model was evaluated by its discrimination and calibration. Internal validation was performed in the primary development cohort. Given the censoring in the development and external validation cohorts (Hippisley-Cox, Coupland & Brindle, 2013; Kalderstam et al., 2013), 1,000-times bootstrapped C-statistics were applied to assess the discrimination of the risk prediction model in the development and external validation cohorts.

Calibration measures how accurately a model’s predictions match the overall observed mortality in the follow-up period. We equally split the observed 5-year mortality into equal deciles across the development and external validation datasets. Calibration performance was assessed with a calibration plot and summarized across the full range of mortality deciles using the Hosmer-Lemeshow statistic. The calibration plot is generated from the means and standard deviation (SD) of the calibration slope and intercept. Furthermore, calibration of the C2ABS2-GLPK score was measured by calibration-in-the-large, which is capable of reporting the difference between the overall observed 5-year mortality and the mean predicted mortality in the development cohort and external validation cohort.

Missing data imputation, cross validation and sensitivity analyses

Imputation for missing variables was considered if fewer than 20% of values were missing. Multiple imputations were implemented to handle missing data with the “mice” package, which is a widely accepted method for population datasets (Azur et al., 2011). Continuous and categorical data were imputed in a 10 sets with a 10-iteration process based on the fitted conditional models until a stopping criterion was satisfied.

Missing data patterns were analysed by the “finalfit” package, and data were considered to be missing at random, which means that each variable in the dataset is equally and randomly likely to be missing and that the conditional likelihood of a missing value is partly dependent on other missing variables, defined by the observed data. Missing at random is in contrast to missing completely at random (Knight et al., 2020).

The linear regression model for continuous variables (mean error rate 15.4%, 95% CI [12.3–16.7%]), logistic regression model for binary variables (mean error rate 6.1%, 95% CI [5.5–7.8%]) and polytomous logistic regression for categorical variables (mean error rate 7.3%, 95% CI [6.5–9.2%]) with more than two levels were applied to the missing parameters to estimate the imputation errors and internally cross-validated errors (Waljee et al., 2013; Knight et al., 2020).

To address the potential effect of missing data imputation, complete case data excluding missing values were analysed to assess the robustness of the C2ABS2-GLPK model to predict 5-year all-cause mortality. Thus, we performed sensitivity analyses by using complete case data to assess the discrimination of the C2ABS2-GLPK model by C-statistics with 1000 iterations of bootstrapping in the development and external validation cohorts, and to compare the results with those of the imputed datasets.

Comparison with existing prognostic scores

The performance of the C2ABS2-GLPK score was compared with that of existing prognostic scores, including the GRACE score, KAMIR score and China Acute Myocardial Infarction (CAMI) score, in the development and the external validation cohorts. The discrimination of the GRACE score, KAMIR score and CAMI score was conducted in the samples with 1,000 iterations of bootstrapping for both cohorts. Differences in the C-statistics were assessed using the DeLong test (Harrell, Lee & Mark, 1996; Steyerberg, 2008; Steyerberg et al., 2010). Moreover, the bootstrapped discrimination of the C2ABS2-GLPK score was also evaluated in the subgroups based on sex, smoking statuses, ST segment elevation myocardial infarction (STEMI), type 2 diabetes, dyslipidaemia (low-density lipoprotein cholesterol [LDL-C] > 2.59 mmol/L), or obesity (body mass index [BMI] > 24) in the development and external validation cohorts.

To evaluate the net benefits of the C2ABS2-GLPK score at the threshold probability for 5-year mortality in the development and external validation sets, we also performed a decision curve analysis to make a clinical judgement about the relative values of mortality (predicting a true positive) and survival rate (predicting a false positive) associated with clinical prediction scores (Vickers & Elkin, 2006). The standardized net benefit was plotted against the threshold probability of considering a patient at high risk of 5-year mortality for the C2ABS2-GLPK score alone and the existing GRACE, KAMIR and CAMI scores applicable to more than 50% of the Chinese AMI patients in the external validation cohort.

Statistical Analysis

Group comparisons were conducted using the chi-squared test for equal proportions or a t tests for normally distributed data; otherwise, the Wilcoxon rank sum tests was used. A p < 0.05 was considered indicative of statistical significance. Statistical analysis was performed using R software (version 3.6.2, R Foundation).

Patients characteristics in the development cohort

A total of 1,471 patients with ICD-9-confirmed AMI between January 2013 and December 2017 from the Third Xiangya Hospital, Changsha city, Hunan Province, were included in the development cohort. During a median follow-up of 5 years (range: 0.08 to 8.75 years), 292 patients (19.9%) died, and 9 (0.6%) were lost to follow-up before the end of the follow-up period. The median age was 64 years (interquartile range (IQR), 55–72), and 71.4% of the patients were men. A total of 173 (11.8%) patients had a history of stroke, 35 (2.4%) experienced preadmission or at-admission cardiac arrest, 844 (57.4%) underwent immediate percutaneous transluminal coronary intervention [PCI], 1,457 (99.0%) were administered statins at admission, and 87 (5.9%) developed acute heart failure ranked as Killip IV at admission. The median N-terminal pro-brain natriuretic peptide [NT-proBNP] level at admission was 1,639 pg/ml (IQR 611–3009), the median serum creatinine at admission was 82 µmol/L (IQR, 68-99), and the median left ventricular end-diastolic diameter at admission was 49 mm (IQR, 45–51). The baseline characteristics and their differences among the survivors, the deceased patients and the censored patients in the cohort are shown in Table 1.

Variable selection and the C2ABS2-GLPK model establishment

Forty-five candidate clinical variables at admission from the patients in the development cohort were included for variable selection. Among these variables, 18 variables were selected in the first-round LASSO regression (Fig. S1, Table S4). The variables with an average OOB value >20% obtained by repeating the random forest algorithm were similar to the variables selected by LASSO regression. The importance of variables, as assessed by the random forest algorithm and reported as the percent increase in MSE (%IncMSE), is shown in Fig. S2 and Table S5. After five rounds of multivariable stepwise backward regression, the 18 LASSO-selected variables were reduced to 10 variables to enter into the final multivariable Cox proportional hazards model (Table S6) and to construct the C2ABS2-GLPK model. The final 10 risk factors, namely, admission or perihospital cardiac arrest, a history of stroke, Killip classification II–IV, fasting blood glucose ≥10 mmol/L, left ventricular end-diastolic diameter (LVDd) ≥60 mm, age ≥70, serum creatinine ≤35 µmol, NT-proBNP ≥300 pg/ml, absence of immediate statin administration or PCI within 24 h after symptom onset, were incorporated into the risk score model for the prediction of 5-year mortality in AMI patients (Table 2).

Construction of the prognostic nomogram and C2ABS2-GLPK score

The nomogram that incorporated the final 10 significant independent factors for predicting the 5-year survival rate in the development cohort was established according to their regression coefficients from the C2ABS2-GLPK model (Fig. 2).

Based on the nomogram, every selected variable was given a continuous score from 0 to 100 corresponding to the regression coefficient. The total C2ABS2-GLPK score was the sum of the scores of each variable. The vertical line was dropped down from the total score row to estimate the relative 1-, 2-, 3-, 4-, and 5-year all-cause survival rates for better application in clinical practice (Fig. 2). The details of the score for each variable are shown in Table S7.

The performance of the C2ABS2-GLPK model and score in the development cohort and internal validation

In the development cohort, the C2ABS2-GLPK model showed good discrimination in predicting 5-year all-cause mortality in patients with AMI, with a C-statistic equal to 0.811 (95% CI [0.786–0.836]) (Table S8). Using the Hosmer-Lemeshow goodness-of-fit test, we found that this model was well calibrated with good coherence between observed and predicted mortality across deciles with the a Hosmer-Lemeshow statistic index of 5.277 (p = 0.728) (Table S8, Fig. 3). The calibration slope was 0.988, and the intercept was 0.716 in the prediction of 5-year mortality using the C2ABS2-GLPK model (Table S9). A slope of approximately 1.0 in the development dataset indicated that the C2ABS2-GLPK model was perfect; however, the intercept was typically far from 0.0, suggesting that the predicted risk was an overestimation of 5-year mortality in AMI patients. Nevertheless, the overall observed mortality vs the mean predicted mortality for the C2ABS2-GLPK score were similar (calibration-in-the-large = 0.02) (Fig. 3).

According to the C2ABS2-GLPK score, AMI patients in the development cohort were stratified into three distinct risk groups by the predicted mortality probabilities following the cut-off points detected by the X-tile analysis. The predicted probability of dying within 5 years was 6.6% in the low-risk group (total points ≤148 points), 15.2% in the medium-risk group (total points, 149–218 points), and 37.3% in the high-risk group (total points >219 points). Compared to that in the low-risk group, the HRs (95% CI) for the medium- and high-risk groups were 2.4 (95% CI [1.6–3.6]) and 7.2 (95% CI [5.1–10.2]), respectively (log-rank p < 0.001) (Fig. 2, Table S10).

The performance of the C2ABS2-GLPK model and score in the external validation cohort

A total of 1,251 patients with ICD-9-confirmed AMI between January 2013 and December 2017 from Xiangtan Central Hospital, Xiangtan city, Hunan Province, were included in the final external validation cohort (Table 1). During a median follow-up of 5 years (range: 0.08 to 8.58 years), 186 patients (14.9%) died, and 65 (5.2%) were lost to follow-up. The median age was 65 years (IQR, 56–73), and 72.8% of subjects were men. A total of 127 (10.2%) patients had a history of stroke, 38 (3.0%) experienced preadmission and at-admission cardiac arrest, 867 (69.3%) underwent immediate PCI, 1,251 (100.0%) were administered statins, and 66 (5.3%) developed acute heart failure ranked as Killip IV at admission. The median NT-proBNP level was 918 pg/ml (IQR, 213–2,587), the median serum creatinine was 79 µmol/L (IQR, 67–95), and the median left ventricular end-diastolic diameter at admission was 49 mm (IQR, 46–51). The C-statistic of the C2ABS2-GLPK score, indicating its discrimination ability, was 0.787 (95% CI, 0.756–0.818) in predicting 5-year all-cause mortality for Chinese AMI patients in the external validation cohort (Table S8). Calibration was also found to be excellent in the external validation cohort: the Hosmer-Lemeshow statistic index for comparison of the observed and predicted risk across deciles of 5-year all-cause mortality in the C2ABS2-GLPK model was 9.495 (p = 0.302) (Fig. 3, Table S8). The slope was 0.976, and the intercept was 0.473 in the prediction of 5-year mortality using the C2ABS2-GLPK model (Table S9). Although a slope nearly equal to 1.0 in the external validation dataset suggested that the C2ABS2-GLPK model was good, the intercept was well above 0.0, indicating that the predicted risk was an overestimation of 5-year mortality in AMI patients. Calibration was also found to be excellent for the C2ABS2-GLPK score: the overall observed mortality vs the mean predicted mortality was nearly equal (calibration-in-the-large = 0.03) (Fig. 3).

The C2ABS2-GLPK score was also applied to the external validation cohort. In the validation cohort, AMI patients could still be divided into the same three risk categories according to the total prediction score. The mean observed 5-year mortality rates in the low-, medium-, and high-risk groups were 8.8%, 19.6%, and 41.7%, respectively. The Kaplan–Meier curves showed that the HRs (95% CIs) for the medium-risk and high-risk categories were 2.6 (1.8–3.6) and 6.5 (4.6–9.1), respectively, compared to that of the low-risk category (log-rank p < 0.001) (Fig. 2, Table S10). The results indicated that a higher C2ABS2-GLPK score was associated with an increased risk of 5-year mortality (p < 0.001).

Comparing the performance of C2ABS2-GLPK score with existing scores

To further validate the superiority in assessing the performance of C2ABS2-GLPK score, we used the c-statistics to compare this score to the GRACE score, KAMIR score and CAMI score in the development cohort and external validation cohort, respectively. The ability of each score was assessed by c-statistic (95% CI). Among them, the c-statistic for C2ABS2-GLPK score either in the development cohort (c-statistic, 0.811; 95% CI [0.786–0.836]) or the external validation cohort (c-statistic, 0.787; 95% CI [0.756–0.818]) were all significantly higher than those of the GRACE score (c-statistic, 0.728; 95% CI [0.697–0.759]) (p < 0.001), KAMIR score (c-statistic, 0.783; 95% CI [0.758–0.808]) (p < 0.001) and CAMI score (c-statistic, 0.558; 95% CI [0.523–0.593]) (p < 0.001), which reflects the higher prognostic value of the C2ABS2-GLPK score for the prediction of 5-year mortality in Chinese patients with AMI (Table S8).

To further assess the clinical utility of the C2ABS2-GLPK score, we also performed decision curve analyses to compare this score with the GRACE score, KAMIR score and CAMI score. The results showed that the C2ABS2-GLPK score had better clinical utility across the threshold of 5-year all-cause mortality applicable to more than 50% of both the development cohort and the external validation cohort (Fig. 4).

Sensitivity analysis

To further confirm the reliability of the C2ABS2-GLPK model, we performed sensitivity analyses by using complete case data. The results showed that the discrimination with the C-statistic based on complete case data was 0.805 (95% CI [0.774–0.836]) in the development cohort and 0.780 (95% CI [0.743–0.817]) in the external validation cohort, which was similar to that with C-statistics of 0.811 (95% CI [0.786–0.836]) and 0.787 (95% CI [0.756–0.818]) in the imputed development and validation datasets, respectively (Table S11).

Subgroup validation

Consistent with the performance in the whole cohort, the predictive performance of the C2ABS2-GLPK score was robust across all subgroups, with C-statistics ranging from 0.770 to 0.832 in subgroups of patients according to sex, history of smoking, types of AMI, and comorbidities of diabetes, obesity and hypercholesterolaemia in the internal and external validation cohorts. Furthermore, the performance of the C2ABS2-GLPK score was superior to that of the GRACE score (c-statistics range: 0.684 to 0.765), the KAMIR score (c-statistics range: 0.744 to 0.807), and the CAMI score (c-statistics range: 0.477 to 0.580) in the indicated subgroups (Table S12, Fig. S3).