Prognosis predictive value of the Oxford Acute Severity of Illness Score for sepsis: a retrospective cohort study

Background The Oxford Acute Severity of Illness Score (OASIS) has shown fair prognosis predictive value in critically ill patients, but its predictive value has not been assessed in septic patients. Objective The aim of this study was to evaluate the performance of the OASIS for the assessment of mortality in septic patients, especially when compared with the Sepsis-related Organ Failure Assessment (SOFA) score. Methods A retrospective cohort study was conducted using data from a public database and septic patients were identified using the Sepsis-3 criteria. The primary outcome was hospital mortality. Data were mainly analyzed using multivariable logistic regression and receiver operating characteristic (ROC) curves. Sensitive analyses were performed in patients with an ICD-9-CM code for sepsis and ROC curves analyses were also conducted in septic patients stratified by the Simplified Acute Physiology Score (SAPS) II as subgroup analyses. Results A total of 10,305 septic patients were included. The OASIS was found to be significantly associated with hospital mortality (odds ratio 1.07 per one-point increase, 95% confidence interval [1.06–1.08]), while ROC curves analyses showed the discriminatory power of the OASIS for hospital mortality was statistically significantly lower than that of the SOFA score (area under the ROC curve: 0.652 vs 0.682, p < 0.001). Results of sensitive analyses were consistent, but the significant difference existed only when the SAPS II was higher than 50 according to results of the subgroup analyses. Conclusions The OASIS might serve as an initial predictor of clinical outcomes for septic patients, but one should be circumspect when it is applied to severer patients.


INTRODUCTION
Scoring systems for outcome prediction have been developed in intensive care medicine for a long time, and some of them have been widely used in the intensive care unit (ICU) (Rapsang & Shyam, 2014). For patients with sepsis, the Sepsis-related Organ Failure Assessment (SOFA) score, which was initially designed to evaluate the severity of organ dysfunction in patients who were critically ill from sepsis (Vincent et al., 1998), has been

METHODS Database
The retrospective cohort study was conducted using data from the Medical Information Mart for Intensive Care III (MIMIC-III) database (v1.4). MIMIC-III is a publicly accessible critical care database which consists of de-identified health-related data about over 40,000 patients stayed in the ICU of Beth Israel Deaconess Medical Center between 2001 and 2012 (Johnson et al., 2016). The access to the database has been approved by the institutional review boards of both Beth Israel Deaconess Medical Center and Massachusetts Institute of Technology Affiliates after completing the CITI (Collaborative Institutional Training Initiative) "Data or Specimens Only Research" course (Qingui Chen (ID: 6533812), Record ID: 24321991).

Patients
All adult patients (age 18 years old) in the database with suspected infection and a SOFA score not less than two points within 24 h after ICU admission were screened for purposes of inclusion, but only those of first hospital admission were considered enrolled. Suspected infection was identified using the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes (See appendix 1 of the report from Angus et al. (2001)). To exclude records of organ donors or potential typographical errors, patients whose length of ICU stay were less than 24 h or whose length of hospital stay were less than length of ICU stay were excluded. Length of ICU stay was determined only by the first ICU stay. No informed consent was required on the de-identified patients.

Data extraction
We extracted data from the database using Transact-SQL and codes from the MIMIC Code Repository (https://github.com/MIT-LCP/mimic-code) (Johnson et al., 2018). Variables extracted included age, sex, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, comorbidities (Steiner, Elixhauser & Schnaier, 2002), and parameters of the OASIS (Table S1), the SOFA score, and the SAPS II. For parameters of these three scoring systems, only data within 24 h after ICU admission were extracted and missing components for calculation were treated as normal (usually zero). Length of ICU stay, length of hospital stay, and the Elixhauser Comorbidity Index (SID30) (Thompson et al., 2015) were also calculated. Since the database had date of birth of patients who are older than 89 years old shifted to exactly 300 years before admission to obscure their age, we corrected them (age-300+ 89) before analyses.

Outcomes
The primary outcome was hospital mortality, and the secondary outcomes were ICU mortality and 28-day mortality after ICU admission. The length of ICU stay and length of hospital stay were calculated only for statistical description. ICU mortality and length of ICU stay were determined by the first ICU stay only.

Statistical analysis
Results were presented as median and 25th-75th percentiles for continuous variables and numbers and percentages for categorical variables unless otherwise stated. Continuous and categorical variables were analyzed using Mann-Whitney U and Chi-square tests, respectively, but Fisher's exact tests were used instead of Chi-squared tests when the expected values in any of the cells of a contingency table are below 10. The associations of OASIS with ICU outcomes were evaluated using multivariable logistic regression or Cox regression analyses. Univariable logistic regression analyses were performed before to examine the determinants of the primary outcome and variables with a p-value less than 0.2 were considered to be included in the multivariable analyses. To examine the linearity assumption of logistic regression, the generalized additive model was used to plot the possibly non-linear relation. Schoenfeld residual plots were also employed to examine the proportional hazard assumption of Cox regression. Discriminatory power was determined by comparing the area under the receiver operating characteristic (ROC) curve (AUC) for each score individually using the method of DeLong, DeLong & Clarke-Pearson (1988). Youden's index was calculated to determine the best threshold of each scoring system and then patients were grouped by the threshold and Kaplan-Meier curves were drawn to visualize their survival. To test the robustness of the results, we performed sensitive analyses in septic patients identified by ICD-9-CM code for sepsis. To examine potential interaction modifiers of the association between OASIS and the primary outcome, logistic regression analyses across different subgroups were performed. For a variable with a p-value for interaction less than 0.05, ROC curve analysis was conducted stratified by the variable after professional judgement. Statistical tests were two-sided when the option was available. A p-value less than 0.05 was considered to indicate statistical significance, but for multiple comparisons of the AUC of the three scoring systems, Bonferroni correction was conducted by adjusting the critical p-value as 0.05/3. Empower (R) (www.empowerstats.com; X&Y solutions, Inc., Boston, MA, USA) and R software, version 3.4.3 (R Core Team, 2016) were used for all statistical analyses.

RESULTS
Characteristics and clinical outcomes of the patients A total of 10,305 patients with sepsis were included finally with a median OASIS of 34 (25th-75th percentiles 28-40). The median age of the subjects was 69 years (25th-75th percentiles 56-80 years) and 5,425 of the 10,305 cases (52.64%) were male. Among them, 5,196 (50.42%) patients required mechanical ventilation on first day and 514 (4.99%) patients required renal replacement therapy on first day. The five most common comorbidities were hypertension (52.95%), fluid and electrolyte disorders (41.49%), congestive heart failure (33.91%), cardiac arrhythmias (33.51%), and deficiency anemia (23.19%). Other characteristics of the patients are presented in Table 1. With regard to their clinical outcomes, the hospital mortality was 12.31% with 1,269 non-survivors and 9,036 survivors, the ICU mortality was 5.84% and the 28-day mortality was 14.91%. The length of ICU stay and hospital stay was 3.48 (25th-75th percentiles 1.98-7.62) and 11.67 (25th-75th percentiles 7.00-19.94) days, respectively.

Association of the OASIS with clinical outcomes of septic patients
As shown in Table 1, non-survivors had significantly higher OASIS on ICU admission than survivors (38 vs 33, p < 0.001). The distributions of the OASIS and the SOFA score with corresponding hospital mortality are presented in Fig. 1. As each score increased, the hospital mortality of the patients approximately increased accordingly. According to the result of univariable logistic regression analyses for variables associated with hospital mortality which was presented in Table S2, age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30) were included in the multivariable regression analyses. Results of curve fitting of the relationship between hospital mortality and several continuous variables were presented in Fig. S1, which indicated that the independent variables were linearly related to the log odds. Schoenfeld residual plots shown in Fig. S2 supported the proportional hazard assumption in Cox models. After adjusting the variables above, the OASIS was significantly associated with hospital mortality (Odds ratio (OR) 1.07 per one-point increase, 95% confidence interval (CI) [1.06-1.08], p < 0.0001), ICU mortality (OR 1.07 per one-point increase, 95% CI [1.06-1.08], p < 0.001), and 28-day mortality (hazard ratio (HR) 1.06 per one-point increase, 95% CI [1.05-1.06], p < 0.001) ( Table 2). Results of analyses of the SOFA score and the SAPS II are presented in Tables S3 and S4, respectively.

Discriminatory power of the OASIS in septic patients
As shown in Fig. 2, the AUC of the OASIS for predicting hospital mortality was 0.652 (95% CI [0.636-0.668]), which was significantly lower than that of the SOFA score (AUC 0.682, 95% CI [0.666-0.699], p < 0.001). The best threshold of OASIS was 34.5 with Table 1 Characteristics and comparison between survivors and non-survivors of all patients.

Variable
All patients (n = 10,305) Survivors (n = 9,036) Non-survivors (n = 1,269) p Results of comparison of the three scores were similar when they were used to predict ICU mortality (Fig. 2). According to the Kaplan-Meier curves presented in Figs. S3, S4 and S5, patients with higher severity scores had shorter survival time regardless of which scoring system was used.

Sensitive analyses
Results of the sensitive analyses were consistent. The OASIS was still significantly associated with hospital mortality (OR = 1.03, 95% CI [1.01-1.05], p < 0.001) and 28-day mortality (HR = 1.03, 95% CI [1.01-1.04], p < 0.001) when patients with an ICD-9-CM code for sepsis were included only, but no significant association was found between the OASIS and ICU mortality (OR = 1.02, 95% CI [1.00-1.04], p = 0.064) (See Table 3). Association of the other two scores with outcomes were presented in Tables S5 and S6,   (Fig. 3).  Notes: Only patients diagnosed as sepsis according to ICD-9 codes were included into the sensitive analysis. Associations of OASIS with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of OASIS with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Statistical significance (p < 0.05) is shown in bold. OASIS, Oxford Acute Severity of Illness Score; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.

Subgroup analyses
Associations of the OASIS with hospital mortality were analyzed using logistic regression models across different subgroups, and the result presented in Table S7 found several variables had a p-value of interaction less than 0.05. After professional judgement, only the SAPS II was chosen as a potential interaction modifier and examined in the subgroup analysis further. As shown in Fig. 4 and Table S8, there was no statistically significant difference in the discriminatory power of the OASIS and the SOFA score for predicting hospital mortality when a SAPS II was lower than 50. In addition, the SOFA score had significantly higher AUC than that of the OASIS for predicting ICU mortality only when a SAPS II was higher than 35.

DISCUSSION
In the study, we evaluated the performance of the OASIS for predicting clinical outcomes of patients with sepsis in ICU. Results of our study indicated that the OASIS on ICU admission was significantly associated with short-term clinical outcomes of patients with sepsis, but its discriminatory abilities of hospital mortality and ICU mortality were statistically significantly lower than that of SAPS II. However, an interesting finding was that there was no statistically significant difference between the mortality discriminatory power of the OASIS and that of the SOFA score when the septic patients were not that severe (evaluated by their SAPS II). Even when the patients were more severe, it should be noticed that the absolute AUC difference between the OASIS and the SOFA score was rather small and there was also an overlap between the 95% CIs of the two scores, so it should be cautious when interpreting the clinical significance of the results. Nevertheless, as far as we know, this is the first study that assesses the prognostic predictive value of the OASIS in patients with sepsis in ICU, which might help clinicians to make good use of this newly developed scoring system. Since the prediction of ICU outcomes might contribute to clinical research and individual patient management (Power & Harrison, 2014), many scoring systems for ICU outcome prediction have been developed and some of them have been routinely used. In terms of sepsis, the most famous scoring system is the SOFA score, an indicator of identifying organ dysfunction and part of the diagnosis criteria of sepsis-3 . The association of the SOFA score with ICU outcomes has been assessed either in mixed ICU patients or septic patients (Ferreira et al., 2001;Jones, Trzeciak & Kline, 2009;Lie et al., 2018), and it has been reported to be a valuable predictor of ICU outcomes. As shown in Table S3 and Fig. 2, results of our study about the SOFA score were consistent. Unlike the SOFA score, the SAPS II is a widely used disease severity scoring system aimed at mixed ICU patients, and it has been reported to perform better for predicting hospital mortality and 90-day mortality than the SOFA score (Granholm et al., 2016). Similar results were also observed in our study that the SAPS II had the best discriminatory power for hospital mortality and ICU mortality among the three scoring systems analyzed. It is not surprising to find the results, since the SAPS II scoring system has more complicated parameters than the other two systems. However, the complexity of the SAPS II scoring system might be a challenge in daily clinical assessment, which impelled us to investigate the performance of the OASIS.
The OASIS system only consists of 10 easily accessible parameters including length of hospital stay prior ICU admission, age, Glasgow Coma Score (Teasdale & Jennett, 1974), heart rate, mean arterial pressure, respiratory rate, temperature, urine output, ventilation, and elective surgery. In fact, the OASIS system was developed from the popular severity scoring systems APACHE IV (Acute Physiology, Age, and Chronic Health Evaluation VI) with the aim of reducing the number of parameters without losing predictive accuracy (Johnson, Kramer & Clifford, 2013). The goal has been achieved to some extent, however, as far as we know, the discriminatory ability of the OASIS for outcomes of septic patients has not been examined especially when compared with the SOFA score, which has similar numbers of items. Before the study we assumed that the OASIS had poorer prognostic predictive value that that of the SOFA score, for the reason that the OASIS does not contain any laboratory parameters. However, surprisingly, we found that the performance of the OASIS was not different from that of the SOFA score when the septic patients were of mild conditions. Although it is impractical to choose and use the OASIS after the SAPS II of a septic patient has been calculated to determine the severity of the patient, we believe that the OASIS is still a promising tool. Clearly, the simple components of the OASIS make it possible to be calculated automatically, which may provide constructive information especially when few laboratory examinations have been done. In addition, in our previous study (Chen et al., 2018) we have found the OASIS might serve as a supplement to the qSOFA score and help to identify more septic patients. Therefore, we reason that perhaps it might also serve as a supplement to the SOFA score to predict outcomes of septic patients. However, further researches are needed to explore the possibilities.
Another issue needed to be discussed is the criteria we used to identify sepsis. In the study, we initially identified septic patients from the MIMIC-3 database using Sepsis-3 criteria. However, this method assumed a causation from infection to organ dysfunction (the SOFA score 2) while in a lot of cases there might be only associations, which might lead to an overestimation of case numbers and an underestimation of mortality (Fleischmann-Struzek et al., 2018). Thus, we then performed sensitive analyses using the ICD-9-CM codes to identify sepsis. Results of the sensitive analyses were consistent, indicating the robustness of our conclusions. However, it should be noticed that the two criteria might represent two distinct populations of sepsis, since the Sepsis-3 criteria was developed in 2016. In our study, the patients identified by the ICD-9-CM codes had higher hospital mortality rate (23.41%) with a median SAPS II of 49.5 and a median SOFA score of 6, so it is not strange to find a lower AUC of the OASIS for hospital mortality in the sensitive analyses since they were more severe.
Several limitations of the study should be noticed. First of all, given the observational nature of our study, selection bias was inevitable although we adjusted a few potential confounders. Second, only the discriminatory power of the OASIS was evaluated, we did not assess its calibration. Third, several variables were not considered. The actual admission years of the patients in the database had been shifted to protect patient confidentiality, so we did not include it in the regression models. Since the database included patients from 2001 to 2012, there might be some differences in outcomes of the patients treated in the early 2000s and later, but we did not take therapy as a covariate into consideration. Last but not least, since all the data in the study was treated as a single dataset, the performance of the OASIS was inherently optimistic.

CONCLUSIONS
To sum up, the retrospective observational study validated significant associations of the OASIS on ICU admission with short-term outcomes of septic patients, and found that although the discriminatory power of the OASIS for hospital mortality was statistically significantly lower than that of the SOFA score, the significant difference existed only when a SAPS II was higher than 50, which suggesting that the OASIS might serve as an initial predictor of clinical outcomes for septic patients, but one should be circumspect when it is applied to more severe patients.