Assessment of three risk evaluation systems for patients aged ≥70 in East China: performance of SinoSCORE, EuroSCORE II and the STS risk evaluation system

Patients

Form January 2004 to September 2016, 2,070 patients from two hospitals (the First Affiliated Hospital of Nanjing Medical University and the East Hospital affiliated to Tongji University) who underwent isolated CABG were invited to participate in the study, which was approved by ethic committees of the two hospitals (Ethical Application Ref: 2017-SR-053; [2017] research (018), respectively). All patients had been discharged when data was extracted. Inclusion criteria were isolated CABG while exclusion criteria were CABG combined with other cardiac surgeries and missing of information. All enrolled patients signed informed consent forms. There were 124 (5.99%) patients excluded from the analysis because of incomplete data (n = 84, 4.01%) and participation in developing SinoSCORE (n = 41,1.98%), and a total of 1,946 procedures constituted made up the database. Each patient’s diagnosis was confirmed by coronary arteriography. According to the study database, the operative risk was predicted by the algorithms online: SinoSCORE is available at http://www.cvs-china.com/sino.asp, EuroSCORE II is available at  http://www.euroscore.org/calc.html and the STS risk evaluation system is available at  http://riskcalc.sts.org/STSWebRiskCalc273/de.aspx. The expected mortality rates of each patient were ascertained by every system above.

The average life expectancy of the Chinese people has reached 76.34 years old, while the mortality rate of CAD has increased significantly in people over 70 years old. To explore the predictive efficacy of the three evaluation systems, in each set, patients were divided into two subsets according to their age: elderly group (age  ≥70, 33.04%, 643/1,946) and younger group (age<70, 66.96%, 1,303/1,946). The calibration and discrimination of the three systems in entire cohort and each subset were assessed and compared. Furthermore, in order to make a fair comparison among the three systems, we compared the expected and observed mortality rates in entire cohort and each subset, respectively.

Outcome endpoint

The outcome of interest in this study was in-hospital mortality, which was defined as postoperative in-hospital all-cause death.

Statistical analysis

Statistical analysis was performed with SPSS version 19.0 (SPSS Inc., Chicago, IL, USA). If continuous variables satisfy the normal distribution, then variables were expressed as mean ± standard deviation, else variables were expressed as median and interquartile range (IQR). Categorical variables were expressed as percentages. Statistical analysis comparing groups was performed using t-tests for continuous variables, Mann–Whitney–Wilcoxon tests for ordinal or continuous variables that do not satisfy the assumptions required for a t-test, and Fisher’s exact or χ² (chi-square) tests for categorical variables. A P-value of less than 0.05 was considered significant.

Performance of the three risk evaluation systems was assessed by comparing the expected and observed in-hospital mortality rates. Calibration (statistical precision) was analyzed by Hosmer–Lemeshow (H-L) goodness-of-fit statistic (Lemeshow & Hosmer Jr, 1982). The H-L statistic measured the differences between expected and observed outcomes. P-value greater than 0.05 means there is no evidence that this risk evaluation system is poorly calibrated. The area under the receiver operating characteristic curve (AUC), which was used to assess how well the evaluated system could discriminate between survivors and non-survivors, describes an estimate of the model’s discrimination ability (Hanley & Mcneil, 1982). The discriminative power of the risk evaluation system is considered excellent if the AUC >0.80, good if  >0.75 and acceptable if  >0.70. Calibration plots of observed versus expected mortality rates for 20 equally sized groups by ranked expected rates calculated of the three systems were constructed. The ideal calibrated predictions consist of a 45°line, Where points below or above the diagonal line indicated overestimation or underestimation respectively. Plots of observed and expected mortality rates of the three systems by age distribution were also drawn, to illustrate the differences mortality rate in subsets.

Finally, the net benefit of three risk evaluation systems for predicting in-hospital mortality was performed by Decision Curve Analysis (DCA). DCA consisted in the subtraction of the proportion of all patients who are false-positive from the proportion who are true-positive, weighting by the relative harm of a false-positive and a false-negative result. DCA was performed using R software version 3.4.0 (R Core Team, 2017) with the package Decision curve.

Performance in elderly group

There were 31 observed deaths, with an observed mortality rate of 4.82% (Table 1). Expected mortality rate of SinoSCORE, EuroSCORE II and the STS risk evaluation system for the elderly group were 1.06(0.87)%, 2.21(1.29)% and 1.27(0.97)%, respectively. The expected ability of the three risk evaluation systems in the elderly group is shown in Table 2. SinoSCORE, EuroSCORE II and the STS risk evaluation system all showed positive calibration in predicting in-hospital mortality (H-L: P = 0.053, P = 0.389 and P = 0.061, respectively) (Table 2). The discriminative power for the elderly group in SinoSCORE was the best (0.829), followed by the STS risk evaluation system (0.730) and EuroSCORE II (0.640) (Fig. 1). SinoSCORE showed excellent discrimination in elderly patients while EuroSCORE II had poor discrimination in elderly patients. The decision curve analysis (DCA) showed the clinical practicability of SinoSCORE, EuroSCORE II and the STS risk evaluation system when the clinician predicted the mortality of patients after cardiac surgery. The results were presented as a graph with the selected probability threshold (i.e., the degree of certitude of postoperative mortality over which patients refused operation) plotted on the abscissa and the net benefits of the risk evaluation system on the ordinate. The net benefit of the SinoSCORE was always greater than that of EuroSCORE II and the STS risk evaluation system regardless of the selected threshold, included between 0 and 20% (Fig. 2).

Performance in younger group

There were 18 observed deaths, with an observed mortality rate of 1.38%. Expected mortality rates of SinoSCORE, EuroSCORE II and the STS risk evaluation system for the younger group were 0.67(0.50)%, 1.18(0.70)% and 0.60(0.50)%, respectively. The expected ability of the three risk evaluation systems in the younger group is shown in Table 2. SinoSCORE, EuroSCORE II and the STS risk evaluation system showed positive calibration (H-L: P = 0.643, P = 0.527 and P = 0.321, respectively) (Table 2). The discriminative power for the younger group in EuroSCORE II was the highest (0.785), followed by the STS risk evaluation system (0.772) and SinoSCORE (0.769) (Table 2, Fig. 1). Three risk evaluation systems all showed good discrimination in the younger group. The decision curve of SinoSCORE, EuroSCORE II and the STS risk evaluation system remained very close regardless of the threshold selected. The benefit of EuroSCORE II was relatively greater than that of SinoSCORE and the STS risk evaluation system: between 0 and 20% (Fig. 2).

Patient baseline data

Of the 1,946 patients, the median age was 66.00(11.00) years, 643 (33.04%) were aged ≥70 years old, 396 (20.35%) were female, and median left ventricular ejection fraction (LVEF) was 63.00 (6.00)%. There were 49 observed deaths, with an overall observed mortality rate of 2.52% (Table 3). The elderly group was more likely to have higher serum creatinine (P < 0.001), chronic obstructive pulmonary disease (COPD) (P = 0.001), The New York Heart Association (NYHA) class IV (P = 0.013), preoperative intra-aortic balloon pump (IABP) (P = 0.040), and hospital mortality (P < 0.001). The younger group had higher weight (P < 0.001), body mass index (P < 0.001), body surface area (P < 0.001), endogenous creatinine clearance rate (P < 0.001), proportionately more grafts (P < 0.001) and number of grafts (P < 0.001) (Table 1).

Expected mortality mean mortality percentage of SinoSCORE, EuroSCORE II and the STS risk evaluation system for the entire cohort were 0.78(0.64)%, 1.43(1.14)% and 0.78(0.77)%, respectively. The expected abilities of the three risk evaluation systems in entire cohort were shown in Table 2. SinoSCORE, EuroSCORE II and the STS risk evaluation system all showed positive calibration in predicting in-hospital mortality (H-L: P = 0.411, P = 0.113 and P = 0.230, respectively) (Table 2). The discriminative power for the entire cohort (AUC) in SinoSCORE was the best (0.829), followed by the STS risk evaluation system (0.790) and EuroSCORE II (0.769) (Fig. 3).

Calibration plots showed that the three risk evaluation systems deviated from the diagonal line, so these risk evaluation systems all underestimated mortality rates in the entire cohort (Fig. 4). When the age factor was considered, the mortality obviously increased with age. SinoSCORE, EuroSCORE II and the STS risk evaluation system overestimated in-hospital mortality rates in patients under 55 years, but underestimated in-hospital mortality rates in patients over 55 years (Fig. 5).