Comparison of 1-year mortality predictions from vendor-supplied versus academic model for cancer patients

Epic end of life care index

The Epic EOLCI uses a logistic regression model on structured EMR data to predict the patient’s risk of 1-year mortality. It was trained on and is designed for use in all adult patients. Model inputs include demographic, lab, comorbidity and medication information. A detailed model brief is available to Epic customers but is not publicly available (Epic Systems, 2020b, 2022). The model was trained on routinely collected EMR data. Death/follow-up data to train the EOLCI was supplied by the clinical sites and it is unknown if high-quality death data sources like the National Death Index were used. The model output score ranges from 0 to 100 with higher score indicating higher mortality risk. Scores of 0–14 are considered low-risk, 15–45 medium-risk, and 45–100 high risk. The vendor stated that the lower threshold for the medium-risk group was chosen such that the sensitivity for 1-year mortality of the medium- to high-risk group was around 50%. The high-risk threshold was set to produce a positive predictive value of 25% for the high-risk group. The vendor reported AUC of 0.89–0.90 for test data from three sites. For cancer patients, the AUC was 0.86. The model brief mentions that the model may not be as well suited to cancer patients as others, because cancer stage is not reliably available as structured data in the EMR so cancer stage is not included as a predictor variable.

The model brief mentions that the score is not meant to be a calibrated prediction of mortality risk, but that the high risk group (score ≥45) had a PPV of 25%. The prevalence of 1-year mortality was around 1.5% in the population used for model development.

Stanford model

We compared the EOLCI to our internally developed model trained in 2020, referred to as the Stanford model. It is a discrete-time survival model for cancer patients deployed for use in patient care since 2020 (Gensheimer et al., 2019, 2021). An earlier version of the model was shown to have approximately doctor-level performance in predicting 1-year mortality (Gensheimer et al., 2021).

The Stanford model was trained in October 2020 with EMR data from 2008 to 2020 from two hospitals and several outpatient sites. Patients with metastatic cancer as identified by diagnosis codes, cancer registry, or cancer staging module in the EMR were included. Only patients with at least one note, lab result, and procedure code were included, to avoid including patients with no care at Stanford. A total of 15,410 patients were included, with an 80%/20% train/test split on the patient level.

For patients included in the Stanford cancer registry (the majority), follow-up/death data from the California Cancer Registry were used. Due to the inclusion of National Death Index data in the California Cancer Registry, nearly all deaths were captured (Eisenstein et al., 2020). For patients not in the cancer registry, EMR and Social Security Administration data were used for follow-up/death information.

The Stanford model has 4,279 predictor variables, including phrases from the text of provider notes and radiology reports, laboratory values, vital signs, and diagnosis and procedure codes. For notes, labs, and vital signs, the most recent one year of data was included; for the other data sources, all past data was included. More recent data was weighted more heavily; for details of the weighting, see a prior article (Gensheimer et al., 2021). For continuous predictor variables like vital signs and lab values, missing values were replaced with the mean value in the training set.

The model is a discrete-time survival model; it predicts the survival probability for each of 12 time intervals from 0 to 5 years of follow-up time. Figure 1 shows the model architecture. It was trained using Keras and the nnet-survival framework (Gensheimer & Narasimhan, 2019). L2 regularization was used to avoid overfitting; the strength was chosen using cross-validation to maximize log likelihood.

For this study we used only the predicted 1-year survival probability from the Stanford model. The prevalence of 1-year mortality in the test set used in development was 84.8%. The C-index for the test set of 3,082 patients used for model development was 0.78; AUC at specific follow-up time points was not calculated.

Patient population

We included adult patients with metastatic cancer who had a return patient visit in a medical oncology clinic at four outpatient sites within Stanford Health Care (Emeryville, Palo Alto, Redwood City, and South Bay) during 2021–2022. These patients were chosen because they had prospective mortality predictions from the Stanford model that were made as part of a quality improvement initiative to increase advance care planning (Gensheimer et al., 2023). Once a week, the Stanford model predictions were generated for patients scheduled in clinic for the upcoming week and automatically saved in a CSV file. Presence of metastatic cancer was determined with the Epic cancer staging module and ICD-10 diagnosis codes.

We had access to EOLCI predictions for all patients in Stanford’s Epic instance from four dates spaced throughout 2021–2022. If a patient had an EOLCI prediction on multiple dates, we only included the earliest eligible date. Patients who had a Stanford model prediction within the week prior to an EOLCI prediction were included in this analysis. We did not have Stanford model predictions at the exact EOLCI prediction dates, since the Stanford model predictions were made once a week. Thus, for each patient at a given EOLCI prediction date, their Stanford model prediction could be from 0 to 6 days in the past. We used the EOLCI prediction date as the starting date for model performance analysis, which would cause a slight bias in favor of the EOLCI since it had slightly more up-to-date information than the Stanford model.

Statistical analysis

Model performance was assessed using Harrell’s C-index, AUC for survival at various time points, Kaplan–Meier curves for quartiles of predicted mortality, and PPV for model-designated high risk patients. For performance analyses at a specific follow-up time point like AUC, patients lost to follow-up prior to that time point were excluded since we did not know if they were alive or dead at the time point of interest. There are pros/cons of various ways of handling such patients (Reps et al., 2021). Date of death was obtained from the EMR and the Social Security Administration Limited Access Death Master File. Details of the linkage to the Social Security Administration data were previously published (Peralta, Desai & Datta, 2022). For patients not known to be deceased, date of last follow-up was determined by finding the last encounter in the EMR. Death/follow-up data were obtained on February 25, 2024.

To help understand why EOLCI and Stanford model predictions differed, we wished to learn which variables were most influential for the Stanford model’s predictions. For this, we used the following procedure, which relied on the fact that the output of the first layer of the neural network is a single value, a “linear predictor” which is a linear function of the predictor variables and coefficients. In a preprocessing step before making predictions, each predictor variable was standardized so that it had mean of 0 and standard deviation of 1 in the training set. For each patient, to find whether a predictor variable’s value led to a longer or shorter predicted survival, we multiplied its value by its coefficient for the first layer of the neural network. To find the predictor variable j increasing survival the most for patient i compared to a “typical” patient with the mean value of j in the training set, the following equation was used (Gensheimer et al., 2021):

$$\begin{array}{c}argmax\\ j\end{array}{\beta }_j\cdot x_{ij}$$where i is the index of the patient, $j\in \left\{1,2,\dots ,n_{features}\right\}$ is the index of the predictor variable, β is the vector of coefficients of the first layer of the neural network, and x_ij is the value of predictor variable j for patient i. The variables decreasing survival the most for each patient were found in an analogous way.

This retrospective study was approved by the Stanford University Institutional Review Board with waiver of consent (#47101). R version 4.2.3 was used for analysis. The article complies with the TRIPOD+AI reporting guidelines (Collins et al., 2024).

Patient characteristics

There were 1,399 patients included. Patient characteristics are in Table 1. Median age at first EOLCI prediction was 67 and 55% were female. Median follow-up was 16.3 months (range 0.03–33). A total of 488 patients (35%) were deceased. Median survival was not reached. By the Kaplan–Meier method, 1- and 2-year survival was 78% (95% CI [76–80]) and 64% [61–67], respectively. A total of 1,283 patients were deceased or had >1 year follow-up and so were evaluable for the 1-year mortality endpoint. Of these, 297 (23%) died within 1 year.

Model performance

Performance metrics are shown in Table 2 and ROC curves are in Fig. 2. AUC for 1-year mortality was 0.73 (95% CI [0.70–0.76]) for the EOLCI and 0.82 [0.80–0.85] for the Stanford model. Results for 180-day mortality were similar. AUC results for demographic subgroups are in Table 3; the Stanford model had better performance than the EOCLI for all subgroups. Kaplan–Meier curves for tertiles of model-predicted mortality are in Fig. 3; there was greater separation between the three groups for the Stanford model.

Of the 1,283 patients evaluable for the 1-year mortality endpoint, 484 had high risk of mortality according to the EOLCI (score 45 or greater). Of EOLCI high-risk patients, 186 (38%) died within 1 year; of EOLCI low-risk patients, 111 (14%) died within 1 year. Thus, for EOLCI high-risk patients, the positive predictive value for 1-year mortality was 0.38, which is higher than the 0.25 reported by Epic in a general patient population. The Stanford model did not have a pre-specified high-risk score cutoff, so we created one by choosing a score cutoff such that 484 patients would be flagged as high-risk (to match the number of EOLCI high-risk patients); this cutoff was >31% predicted probability of 1-year mortality. At this cutoff, the Stanford model’s positive predictive value for high-risk patients was 0.65. The diagnostic odds ratio for the EOCLI was 3.87 and for the Stanford model it was 8.10. This measure is the ratio of the odds of the patient being predicted to die within 1 year if they truly died within 1 year, relative to the odds of the patient being predicted to die within 1 year if they survived >1 year. We also calculated the Gini index for each model; this is reported in the Supplemental Results section (Wu & Lee, 2014).

For metrics at a specific follow-up time point such as AUC and positive predictive value, patients lost to follow-up prior to the follow-up time point of interest were excluded. As a sensitivity analysis, we re-calculated these performance metrics without excluding such patients but instead counting them as alive (reported in Supplemental Results section). The Stanford model performance was still superior to the EOLCI, though its positive predictive value was lower than the main analysis at 0.40.

To get insight into why the Stanford model’s performance was higher in this evaluation, for each patient we recorded the Stanford model’s five predictor variables that had the most positive influence on predicted survival time, and the five variables with the most negative influence. The most common of these high-influence variables are in Table 4. The EOLCI’s predictor variables are not publicly available so we are not able to include a full comparison with them. Some high-influence variables in the Stanford model are also predictors for the EOLCI, such as age and albumin, and some are not, such as denosumab and granulocyte count.