Deep-learning tool for early identification of non-traumatic intracranial hemorrhage etiology and application in clinical diagnostics based on computed tomography (CT) scans

Introduction

Spontaneous and nontraumatic intracerebral hemorrhage (ICH) represents a significant global health concern, with an annual incidence of 10–30 per 100,000 population. As the most devastating stroke subtype, ICH accounts for 15−20% of all strokes and is associated with substantial morbidity and mortality worldwide (Qureshi, Mendelow & Hanley, 2009; Labovitz et al., 2005). While hypertension is the predominant cause of ICH, a considerable number of cases stem from underlying macrovascular abnormalities, including arteriovenous malformation (AVM), aneurysm, cavernous malformation (CM), and Moyamoya disease (MMD) (Qureshi, Mendelow & Hanley, 2009; Meretoja et al., 2012; Zhao et al., 2023). Timely and accurate diagnosis, coupled with appropriate surgical interventions, can significantly reduce mortality and improve functional outcomes in ICH patients (van Donkelaar et al., 2020). Thus, rapid and precise identification of ICH etiology is crucial, as treatment strategies vary considerably depending on the underlying cause, particularly for vascular abnormalities (Hemphill et al., 2015; Derdeyn et al., 2017).

The optimal approach for early identification of macrovascular causes in non-traumatic ICH patients remains a subject of debate (Cordonnier et al., 2018). Although digital subtraction angiography (DSA) was the gold standard for detecting macrovascular abnormalities, its invasive nature poses inherent risks. Non-contrast computed tomography (NCCT) has emerged as the primary diagnostic tool in emergency settings for patients presenting with symptoms suggestive of hemorrhage, owing to its widespread availability, cost-effectiveness, and rapid acquisition capabilities (Hemphill et al., 2015). Following ICH identification via NCCT, computed tomography angiography (CTA) and magnetic resonance angiography (MRA) are typically recommended for detecting underlying vascular lesions (Hemphill et al., 2015). Early diagnosis of these vascular abnormalities can significantly impact clinical management and prognostic predictions in ICH patients. However, the high costs and limited accessibility of time-intensive magnetic resonance imaging (MRI), MRA, and CTA often restrict their application in the acute phase (Demchuk, Menon & Goyal, 2016). In clinical practice, particularly in underserved healthcare environments, it’s rarely feasible for every ICH case to undergo emergency CTA or MRA examination (Bekelis et al., 2012). Moreover, there’s a lack of clear guidelines for selecting patients for further angiographic imaging to identify etiology (Hilkens et al., 2018; Cordonnier et al., 2010).

While NCCT can potentially screen for macrovascular causes, identifying key features remains challenging. Enhancing the sensitivity, specificity, and accuracy of NCCT screening, particularly in primary etiology identification of ICH, could prove invaluable in assisting physicians to make rapid, well-informed decisions regarding further angiographic imaging or immediate intervention (Singh & Bhatia, 2019). Recent advancements in artificial intelligence, specifically convolutional neural networks (CNNs), have demonstrated excellent performance in various clinical image-based recognition tasks, showing promise as a diagnostic strategy (Ye et al., 2019). Several deep learning algorithms based on CNN models have received approval from the US Food and Drug Administration (FDA) for medical image interpretation, further underscoring their potential in this field (Ye et al., 2019). Given the potentially less favorable outcomes that from delayed clinical management due to ICH etiology identification process, such as vasospasm and delayed cerebral ischemia. An accurate and timely deep learning model that could help clinicians reliably identify ICH etiology from NCCT scans is especially valuable. However, to date, the integration of human and artificial intelligence (AI) for ICH etiology detection have barely started, and the ability of deep learning systems to augment clinician performance remains relatively unexplored. In the past, the subtype of ICH was predicted based on the 2D CNN model. Ye et al. (2019) tried to apply a 3D CNN-based approach to detect ICH, using a simple CNN network with five convolutional layers and two fully connected layers. The performance of this plain 3D CNN seems to improve. It is unclear whether this method can produce reliable predictions (Ye et al., 2019).

This study aimed to investigate the potential of a CNN system for diagnosis of ICH etiology from NCCT scans. The main hypothesis of our research is that a deep learning model can achieve high accuracy in classifying different ICH etiologies and provide significant diagnostic support to clinicians in a real-world setting. In addition, we also compared the performance of the proposed system with clinician diagnoses. We further conducted a simulated study to compare the clinicians’ performance with and without the deep learning system complements.

Materials and Methods

Datasets and clinical taxonomy

Data collection

For dataset development, we conducted a retrospective review of NCCT scans from 4,019 consecutive patients with non-traumatic acute ICH who presented at Beijing Tiantan Hospital between January 2011 and April 2018 (Fig. 1). Two radiologists independently confirmed the ICH diagnoses (Meretoja et al., 2012). We included only the initial NCCT scan following each ICH event in our analysis. Data were collected similarly as previously described by Meretoja et al. (2012) We also uploaded our detailed methods to Protocol.io (dx.doi.org/10.17504/protocols.io.yxmvmejoog3p/v1). Portions of this text were previously published as part of a preprint (Zhao et al., 2023).

We excluded examinations that were: (1) diagnosed as traumatic, (2) associated with a history of brain surgery or external ventricular drain, (3) conducted 24 h or more after the first ICH event, (4) duplicated scans with rescan due to compromised image quality, initial head motion and metal artifacts arisen from finger and ear rings, (5) lacked a complete image series, or (6) were from patients under 4 years of age. We reviewed medical records for patient age, sex, complaint, medical history, known hypertension, and impaired coagulation (Meretoja et al., 2012). Of the original 4,019 records, 1,868 were eligible for the model development dataset.

To assess the accuracy and generalizability of our developed system, we collected two independent test datasets: NCCT scans of consecutive patients with non-traumatic ICH who presented at Beijing Tiantan Hospital from April 2018 to November 2018 (TT200), and NCCT scans of consecutive patients with non-traumatic ICH who presented at the Affiliated Hospital of Shandong Jining Medical College from April 2018 to January 2019 (SD98). The NCCT parameters were 120 kVp, 300 mA, 512 × 512 image matrix, 1-s rotation, 5-mm section thickness, 5-mm intersection space, and CTDIvol was approximately 40 mGy. We applied the same exclusion criteria, preprocessing, and labelling methods to both datasets as we did to the development dataset (Meretoja et al., 2012).

This study received approval from the Institutional Review Boards of the Beijing Tiantan Hospital and Affiliated Hospital of Jining Medical College (KYSQ 2019-163-01). All medical images and clinical data underwent complete anonymization (Meretoja et al., 2012). We retrieved Digital Imaging and Communications in Medicine (DICOM) images from picture archiving and communication system servers in compliance with the Health Insurance Portability and Accountability Act. To protect patient privacy, we removed all protected health information from the dataset, including names, dates of birth, medical record numbers, and other direct identifiers. We obtained written consent from all participants.

Data labelling

Our etiologic classification and definitions were based on the method proposed by Meretoja et al. (2012). We confirmed etiology diagnoses using NCCT scans in conjunction with additional clinical evidence, including CTA, MRA, digital subtraction angiography, or surgical pathology notes. Two radiologists with over 10 years of experience labeled the ICH etiologies, with any disagreements resolved by a third investigator (Meretoja et al., 2012). We categorized each NCCT case as having one of the following six causes: hypertensive hemorrhage, aneurysm, AVM, MMD, CM, or other causes, which included cerebral amyloid angiopathy, systemic disease, undetermined causes, arteriovenous fistula.

Results

For model development, we retrospectively reviewed the NCCT scans consecutive patients with non-traumatic acute ICH who presented at Beijing Tiantan Hospital between January 2011 and April 2018. Of 4,019 NCCT scans reviewed, 1,868 were eligible for inclusion, as shown in Fig. 1. For a qualified input NCCT DICOM volume of an ICH patient, the system was designed as an end-to-end approach, including skull-stripping, rotation and classification, that analyzed the entire NCCT volume automatically and produced a series of scores representing the probabilities of different etiologies. The TT1868 dataset was randomly divided into five folds with four folds in the training set and one-fold in the validation set; TT200 and SD98 were prepared as test sets (Fig. S2). The patient demographics and image characteristics from the 1,868 records used for model development are summarized in Table 1.

In our analysis of 1,868 ICH scans, we identified 628 (33.6%) as hypertension-related, 845 (45.2%) as aneurysm-related, 44 (2.4%) as MMD-related, 34 (1.8%) as CM-related, 104 (5.6%) as AVM-related, and 213 (11.4%) as related to other causes. Following pre-processing and data complementation, our dataset expanded to 33,624 NCCT volumes.

The TT200 dataset comprised 200 examinations: 75 hypertension-related, 70 aneurysm-related, 12 MMD-related, 11 AVM-related, 14 CM-related, and 18 related to other causes. The SD98 dataset included 61 hypertension-related, 25 aneurysm-related, five MMD-related, four AVM-related, one CM-related, and two other cause-related records (Table 1).

Tables S1 and S2 present the comparative results of various CNN models for model selection. Table 2 and Fig. 2 summarize the performance of the ICHNet algorithm at selected operating points. On the TT200 dataset, ICHNet achieved AUCs of 0.986 (95% CI [0.967–1.000]) for aneurysms, 0.952 (95% CI [0.917–0.987]) for hypertensive hemorrhage, 0.950 (95% CI [0.860–1.000]) for AVM, 0.749 (95% CI [0.586–0.912]) for MMD, 0.837 (95% CI [0.704–0.969]) for CMs, and 0.839 (95% CI [0.722–0.959]) for other causes. On the SD98 dataset, ICHNet achieved AUCs of 0.945 (95% CI [0.882–1.000]) for aneurysms and 0.883 (95% CI [0.818–0.948]) for hypertensive hemorrhage.

Six raters evaluated the TT200 test dataset to assess the deep learning system’s performance. In task one, based solely on image information, four clinicians achieved an average accuracy of 0.706, compared to the proposed approach’s accuracy of 0.760. Bootstrapping tests indicated that our system significantly outperformed three of four clinicians (p < 0.05), as shown in Table S3 and Fig. S3.

In task two, with additional clinical information provided, raters achieved an average accuracy of 0.725. Despite this improvement, our approach still performed significantly better than three of six raters (p < 0.05) (Fig. S4 and Table S3).

In task three, with access to the algorithm’s probability predictions, significant increases were observed in mean sensitivity to aneurysms (0.091, p < 0.05) and hypertensive hemorrhage (0.133, p < 0.05), and mean specificity to AVM (0.032, p < 0.05), MMD (0.044, p < 0.05), and CM (0.028, p < 0.05) with AI complements on NCCT images and clinical information. Table 3 and Fig. 3 present these results. Table S4 and Fig. S5 detail each rater’s performance. The mean accuracy of raters’ diagnoses significantly improved from 0.725 to 0.803 (p < 0.01) with deep learning complements (Fig. S6).

Figure 4 presents the evaluated Cohen’s kappa coefficients for pair-wise concordance. Pre-complement, these coefficients ranged from 0.51 to 0.69 across all rater pairs. Post-complement, concordance improved to 0.83 (p < 0.01), with nine of 15 pairs showing excellent agreement (>0.75) (Fleiss, Levin & Paik, 2013). The Fleiss’ kappa for overall rater concordance increased from 0.61 to 0.75 (p < 0.01) after deep learning system complements.

Discussion

This study presents a pioneering deep learning system trained to diagnose ICH etiology from NCCT scans, achieving accuracy comparable to human experts. We validated the algorithm on two independent prospective datasets comprising diverse cases. Notably, when clinicians utilized the model’s output, their diagnostic accuracy for ICH etiology significantly improved, surpassing the performance of the algorithm alone.

The field of AI in medical imaging has shown promising results, potentially enabling rapid, cost-effective, and accurate diagnostics with global accessibility. Deep learning algorithms have been successfully applied to various medical imaging modalities, including OCT scans (De Fauw et al., 2018), retinal fundus images (Gulshan et al., 2016), and digitized pathology slides (Golden, 2017). Previous studies have developed algorithms for identifying head computed tomography (CT) scan abnormalities requiring urgent attention (Chilamkurthy et al., 2018) and for detecting acute ICH and classifying its location from NCCT (Lee et al., 2019).

However, while progress has been made in applying deep learning to ICH CT imaging interpretation, identifying location alone is insufficient for clinical practice. Several prediction scores have been developed to assess the risk of underlying macro-vascular etiology in non-traumatic ICH patients, such as the simple ICH score, the secondary ICH score, and the DIAGRAM prediction score (Hilkens et al., 2018). Yet, these scores cannot predict or classify the specific ICH cause. Our system, capable of predicting various ICH etiologies, could provide valuable information to clinicians for triage and imaging selection (Singh & Bhatia, 2019).

CT-angiography, although considered a standard diagnostic tool for ICH etiology, is not routinely performed in most centers (Becker et al., 1999). Studies on spontaneous ICH indicate that only 37.4–76.0% of ICH patients undergo CTA scans (Bekelis et al., 2012; Oleinik et al., 2009; Sorimachi et al., 2020). A large study involving 1,423 consecutive adult ICH patients reported CTA sensitivity of 95.7% for secondary ICH diagnosis (Sorimachi et al., 2020), with sensitivities of 99.1% and 90.4% for aneurysm and AVM diagnosis, respectively. Our algorithm demonstrated comparable performance on two independent datasets. At a 90% specificity threshold, our model’s sensitivities in the TT200 dataset were 97.1% and 90.9% for aneurysm and AVM diagnosis, respectively, rivaling or potentially surpassing CTA performance.

To simulate a typical clinical setting, we provided raters with clinical information in the second task. Although this improved the sensitivity and specificity of clinicians’ diagnoses, they generally remained less accurate than our proposed algorithm. Figure S8 illustrates cases where raters’ diagnostic accuracy improved after considering the system’s predicted probabilities. Figure S7 displays spatial resolution histograms for the training and test datasets, showing that the TT200 dataset’s resolution distributions more closely resembled the training dataset (TT1868) than SD98.

When clinicians had access to the model’s predicted probabilities, they achieved higher specificity in diagnosing AVM, MMD, and CM, while maintaining high sensitivity for AVM and MMD diagnoses. This is particularly crucial as false negatives in macrovascular etiology diagnoses should be minimized in clinical practice. Notably, the sensitivity of clinicians’ diagnoses for hypertensive hemorrhage, the most common ICH etiology, increased significantly with AI assistance.

Our results demonstrate the model’s high accuracy in classifying ICH etiologies, as evidenced by the impressive AUC values across various categories. The model’s performance, particularly in challenging cases, underscores its potential to enhance clinical decision-making and optimize patient management strategies. It’s crucial to emphasize that this deep learning system should be viewed as a supportive tool for clinicians rather than a standalone diagnostic device. Overreliance on AI systems can lead to issues such as automation bias or deskilling of clinicians. Therefore, we envision this tool as augmenting clinical decision-making, not replacing it.

This study had some limitations. Our study was hospital-based, and the two datasets were from two large tertiary referral hospitals. The distribution of the hemorrhage etiologies was unbalanced and may have been biased towards referrals. Despite the substantial number of scans with diversified etiologies, the number of cases in minority classes, like MMD and CM, were limited. While our results are promising, we acknowledge that clinical implementation of this system would require careful consideration and further validation. Future research should focus on developing and validating a protocol for integrating this tool into clinical workflows, with particular attention to patient safety and the potential for unintended consequences. Therefore, it is important to enrich the training database in the future work, especially the minority classes. Spectral CT provides enhanced tissue characterization and improved contrast resolution, which could potentially augment the performance of our deep learning model (Sedaghat et al., 2021). Future investigations could explore how incorporating spectral CT data into our model might improve diagnostic accuracy and provide additional insights into various intracranial hemorrhage etiologies. Future work should also focus on developing explainable AI techniques for this system. Providing interpretable explanations for the model’s predictions could enhance clinician trust, facilitate error detection, and potentially offer new insights into radiological features associated with different ICH etiologies.

Conclusions

In conclusion, we presented a novel deep learning system that analyzes clinical NCCT scans and makes predictions of ICH etiology with sensitivities, specificities, and accuracies similar to those of clinical specialists. This system could potentially assist in triaging patients with ICH for further neurovascular evaluation, particularly in cases where the system indicates a high probability of etiologies such as aneurysm or AVM that would require immediate attention. We also demonstrated that integration of the deep learning model can augment clinician performance and could equip specialists with the ability to make better decisions. Further research is necessary to develop and validate a protocol for safely integrating this algorithm into clinical practice before assessing its potential impact on care outcomes.

Supplemental Information