Enhancing heatstroke prediction accuracy with interpretable machine learning: a multi-center data-driven approach

Introduction

Global climate change has caused a dramatic rise in air temperatures, leading to an increased incidence of heat-related illnesses and posing a significant threat to human health (Gauer & Meyers, 2019; Shimazaki et al., 2020). Heatstroke is a severe clinical syndrome and medical emergency, which is considered as the most severe form of heat-related illnesses (Hoek et al., 2023; Bouchama et al., 2022; Asmara, 2020). It is marked by a core temperature with exceeding 40 °C and abnormal manifestations of central nervous system. Additionally, heatstroke can easily progress to disseminated intravascular coagulation and multi-organ dysfunction, both of which typically result in poor prognosis. Prior researches have demonstrated that the mortality rate of heatstroke can reach 40–50% (Epstein & Yanovich, 2019; Glaser et al., 2016). Thus, social attention to heatstroke has been growing year by year and accurate prediction of heatstroke is becoming essential.

It is regrettable that the diagnosis of heatstroke continues to be heavily dependent on core body temperature and central nervous system symptoms (Lin et al., 2024). This conventional diagnostic approach may lead to misdiagnoses in certain cases. Previous study has reported that the patient admitted to a hospital with elevated biochemical indicators and consciousness disorder but normal body temperature was diagnosed with heatstroke (Yeoh & Law, 2023). As a result, several predictive models have been developed to identify heat-related illnesses, including heatstroke (Wei et al., 2022; Wan et al., 2023). However, these models have not been widely implemented in clinical settings due to the absence of comprehensive data and sophisticated algorithms. There is an urgent need for a novel diagnostic method that can accurately diagnose heatstroke, thereby enhancing diagnostic accuracy and reducing the rate of missed diagnoses. Recently, predictive models that integrate machine learning (ML) algorithms and extensive clinical data have attracted significant attention. ML predictive models for heatstroke and its prognosis demonstrate exceptional performance and generalizability (Wang et al., 2019, 2023). Nonetheless, the inherent opacity of ML algorithms presents a challenge to their clinical application (Watson et al., 2019). At present, algorithms designed to elucidate ML models are emerging. We are optimistic that these explanatory algorithms will enable a deeper comprehension of the underlying decision-making processes within ML models.

In this study, we developed and validated an ML model for heatstroke prediction using a multi-center database. Additionally, we employed the SHapley Additive exPlanations (SHAP) method to interpret the outputs of the optimal ML model.

Methods

Data collection and population

In this multi-center retrospective study, 24 hospitals across China provided data from 2021 to 2023 in Excel sheet format. The training set comprises heat-related disease data from 2021 and 2022, while the 2023 data serve as the testing set (Fig. 1). The criteria for inclusion were: (1) age ≥18 years, (2) patients admitted to the hospital with a confirmed diagnosis of heat-related illness (Bouchama & Knochel, 2002). The following were excluded: (1) patients younger than 18 years, (2) those with congenital coagulopathy or severe chronic liver or kidney disease, (3) more than 30% missing data. This study was approved by the ethics committee of the 908th Hospital of Chinese PLA Logistic Support Forces, which waived the requirement for informed consent because of the study’s retrospective design (approval document ID 908yyLL028).

Results

Building and calibrating machine learning models for heatstroke

We compiled a total of 23 clinical and biological features, and through applying recursive feature elimination, six features for heatstroke were ultimately selected. They were TT, CKMB, Lac, core temperature, mean arterial pressure (MAP) and D-dimer. Table 2 and Fig. 2 summarize the performance of the ML models. In the training dataset, the GBM model exhibited the highest performance, achieving an area under the receiver operating characteristic curve (AUROC) of 0.971. In comparison, the performance metrics for KNN, NN, MLP, SVM, GP, NB, XGB, and LR were 0.867, 0.710, 0.772, 0.672, 0.702, 0.770, 0.789, and 0.594, respectively. The testing datasets was employed to confirm the model’s effectiveness in predicting heatstroke, and GBM model showed good performance with an AUROC up to 0.836 (Fig. 2). We also incorporated area under the curve (AUC), accuracy, balanced accuracy, kappa, sensitivity, specificity, precision, and F1-scores for a comprehensive evaluation of the model’s performance. Those of GBM were higher than other models (Table 2). The calibration curves of the GBM model in the validation set exhibited robust concordance between observed and predicted probabilities, with a calibration intercept of 0.15 (95% confidence interval (CI) [−0.17 to 0.48]) and a calibration slope of 1.23 (95% CI [0.86–1.60]) (Fig. 3).

Table 2:

Performance of ML models for predicting heatstroke.

Model	AUC	Accuracy	Balanced accuracy	Precision	F1-scores	Sensitivity	Specificity
GBM	0.966	0.879	0.858	0.938	0.833	0.750	0.966
SVM	0.672	0.671	0.547	0.679	0.802	0.887	0.981
XGB	0.789	0.702	0.605	0.723	0.610	0.733	0.943
MLP	0.772	0.726	0.643	0.746	0.521	0.648	0.934
KNN	0.867	0.775	0.707	0.822	0.699	0.528	0.943
NN	0.710	0.663	0.582	0.553	0.615	0.705	0.868
GP	0.702	0.700	0.587	0.783	0.676	0.795	0.968
LR	0.594	0.655	0.533	0.594	0.182	0.109	0.959
NB	0.770	0.722	0.636	0.747	0.517	0.665	0.937

DOI: 10.7717/peerj.20377/table-2

Interpretability and clinical benefit analysis

Finally, we utilized the SHAP method to calculate and display feature importance. Figure 4 provides an extensive overview of how importance is distributed among all features, making it easier to understand the significance of each feature. TT, CKMB, Lac, core temperature, MAP and D-dimer were confirmed as the top six influencing variables for heatstroke. CKMB is the most influential factor in the model predictions. Figure 5 depicts the clinical effectiveness of ML models across different risk thresholds. The GBM model showed a significant advantage in decision curve analysis (DCA) compared to the “treat-all” and “treat-none” strategies when the threshold probability exceeded 20%. Figure 6 showed the individual force plot for three patients. The force plot primarily highlighted the role of each variable in predicting heatstroke for an individual. Variables depicted in red increase the likelihood of heatstroke, whereas those in blue decrease the heatstroke risk.

Discussion

The integration of machine learning into medical and clinical settings represents a significant emerging research trend (Haug & Drazen, 2023; Van Calster & Wynants, 2019). In this study, we built and verified a heatstroke prediction model by employing nine machine learning algorithms. This model was built using routine clinical parameters and laboratory variables collected from 24 hospitals. The results indicated that GBM model had the highest AUC value (AUC > 0.8) in both the training and testing datasets, demonstrating the better discriminatory performance. Additionally, other metrics such as accuracy, balanced accuracy, sensitivity, specificity, precision, and F1-scores demonstrated superior performance compared to other machine learning models, highlighting the efficacy of the top-performing model. Decision curve analysis revealed that the GBM model could increase the net benefit for predicting heatstroke. Illustrating the threshold ranges above the prediction-all and -none curves provides insight into how the model can be utilized in real-world clinical settings. Therefore, compared with other ML models, it had better predictability to identify heatstroke in clinical settings. Doctors can readily assess patients’ risk of heatstroke and take proactive measures to prevent it. Additionally, we anticipate that this prediction model could aid in optimizing medical resources by facilitating the early detection and monitoring of high-risk heatstroke patients. However, further investigations are required to validate these assumptions.

Numerous models have been developed to predict heat-related illnesses, including heatstroke, by integrating various weather data (Wan et al., 2023; Takada et al., 2023). These models exhibit relatively effective prediction capabilities for heat-related illnesses or heatstroke. However, they failed to take into account clinical and laboratory variables, and some were restricted to daily-unit predictions. Consequently, weather data may not be sufficient to accurately predict heatstroke in patients, and a new heatstroke prediction model is needed. A recently developed interpretable machine learning model to predict heatstroke has been found to be effective (Ogata et al., 2021), but this model still requires further validation and merely include weather relevant variables. In addition, the model based on Japanese society may not be suitable to patients in China because of regional and ethnic differences. Hence, the objective of our study was to build machine learning models specifically for screening heatstroke in the Chinese population.

There has been widespread application of explainable machine learning algorithms to predict the occurrence or outcome of various diseases (Guan et al., 2023; Nielsen et al., 2024; Deshmukh & Merchant, 2020; Choi et al., 2023). We implemented an explainable analysis to describe how these features influenced the GBM model and how this model achieved individual case predictions. Prior studies showed that machine learning models were highly predictive but poorly interpretable (Wan et al., 2023; Van Calster & Wynants, 2019; Lundberg & Lee, 2017). When users entered data to acquire outputs, it was hard to understand how the model generated predictions. The lack of transparency in ML models greatly restricts their clinical applicability, particularly in complicated cases with significant medical consequences. The explainable ML model we constructed enable users to better comprehend the model’s decision-making process, thus enhancing its reliability and transparency. In terms of the number of variables, Ogata et al. (2021) and Wan et al. (2023) employed 18 and 33 features, respectively, which may be susceptible to collinearity, potentially biasing the model’s predictions of heatstroke. Simultaneously, when a prediction model includes a large number of features, its complexity and time-consuming nature make it challenging for clinical physicians to use. We employed recursive feature elimination to identify the optimal features and ultimately chose six indicators for developing ML models. This approach helps mitigate potential biases that can arise from manually selecting predictors without sufficient experience (Hayashida et al., 2018; Staartjes et al., 2022). Through the interpretative analysis, we discovered that CKMB was the key predictor for the GBM model’s decision-making process. Other essential predictors included MAP, D-dimer, lactate, TT, and core temperature. These early indicators may be associated with subsequent diagnostic criteria for heat-related illnesses. To ensure the integrity of the predictive modeling, this study restricted all variables to data collected during the pre-diagnosis admission phase, effectively preventing label leakage that could occur if diagnostic criteria were retroactively incorporated into the predictors. The integration of these common clinical features into machine learning models demonstrates promising potential for practical clinical applications. Moreover, decision-making process of the GBM model was further demonstrated by SHAP summary and force plot, adding clinical credibility and interpretability.

This study makes several significant contributions, including the employment of various sophisticated ML techniques for modeling, the application of the SHAP method for interpreting the decision-making mechanism employed by the ML model, and the use of decision curve analysis. The inclusion of 691 patients from 24 medical centers enhances the model’s generalizability. Nevertheless, there are some limitations to the study. Firstly, it is retrospective and may inherently contain biases. Second, it lacks the design of clinical trials to evaluate the model’s performance in real-world settings. Thirdly, we used the KNN method to generate interpolation values, which could lead to estimation errors. Finally, the study cohort consisted exclusively of patients with confirmed heat-related illnesses, with a predominance of males, which limits the broad applicability of the model across different gender groups. Future research should prospectively evaluate the model’s performance in real-world settings and explore whether this prediction model can effectively reduce cases of heatstroke.

Conclusion

An ML model could serve as a reliable tool for heatstroke identification and possess robust predictive power, assisting doctors in recognizing patients at high risk. This would facilitate early intervention and the effective prevention and management of individuals at high risk of heatstroke.

Supplemental Information