Modeling using clinical examination indicators predicts interstitial lung disease among patients with rheumatoid arthritis

Ethical statement

This study is a retrospective study that was approved by the Ethics Committee of Harbin Medical University (HMU) (Approval number: HMUIRB20150028) and was carried out in accordance with the Declaration of Helsinki.

Patients

The electronic medical record systems of the first and second affiliated hospitals of HMU were used to access the medical records of hospitalized patients that were clinically diagnosed with RA For each patient, high-resolution computed tomography (HRCT, Discovery CT 750 HD; GE Medical Systems, LLC., Waukesha, WI, USA) was performed to examine whether the complication of ILD or PF was present. In the second affiliated hospital of HMU, 32 CEIs were retrieved from the hospital’s electronic system of medical records (form January 1, 2013 to December 31, 2015). All the CEIs were categorized into three classes. The three classes were basic information (two items), routine blood test (22 items), and routine urine test (eight items). In the first affiliated hospital of HMU, four CEIs including age, eosinophil count (EO), platelet count (PLT), and white blood cell count (WBC) were retrieved from the hospital’s electronic system of medical records (from January 1, 2014 to December 31, 2015). If a patient’s clinical examination was inadequate, he or she was not taken into consideration for inclusion as a subject. It should be especially noted that there were no other reasons for rejecting subject inclusion, such as age or gender.

Identification of ILD-associated CEIs

The software MedCalc v15.8 (MedCalc, Mariakerke, Belgium) was used to perform receiver operating characteristic (ROC) curve analysis and calculate the Youden index for each CEI. The Youden index was the sum of sensitivity and specificity minus 1 as defined previously (Youden, 1950). This effort aimed to investigate whether a CEI can be used as a marker to distinguish patients suffering from ILD from those with healthy lungs. It should be noted that only the RA patients with healthy lungs and those suffering from ILD were included in the calculation of the Youden index and the subsequent construction of networks and models. ILD-associated CEIs were identified only when the area under the ROC curve (AUC) was significantly larger than 0.5 (P < 0.05), and were retained for further integration by ANN.

Data preprocessing and model integration of ILD-associated CEIs

ILD-associated CEIs were normalized into a 0–1 number before further analysis as previously described (Zhu & Kan, 2014). The Intelligent Problem Solver (IPS) tool in the software STATISTICA Neural Networks (SNN, Release 4.0E; Statsoft, Tulsa, OK, USA) was applied to construct a radial basis function (RBF)-ANN model to investigate the effect of CEI integration on ILD association. In this study, the model was simply named as ANN I. The model output then underwent normalization processing and Youden index calculation to investigate whether an obvious association with ILD status was still present after CEI integration. The holdout cross-validation method was applied for preliminary validation of the model as IPS randomly divided the patients into three subsets (training set, verification set, and testing set) in a 2:1:1 ratio. Thus, one-quarter of all the patients did not participate in model building and were used for model testing. The IPS calculated correlation coefficients for the training set (R_Tr) and the testing set (R_Te). The two correlation coefficients measured the correlation between model output and status of ILD. Similar values of R_Tr and R_Te indicates good generalization ability of the model.

Euclidean distance calculation and construction of patient–patient similarity network

For any two patients, we calculated their Euclidean distance in an n-dimensional space, in cases in which ILD-associated CEIs were re-defined as space coordinates. The value of n was the sum of ILD-associated CEIs. Following the clustering algorithm proposed by Rodriguez & Laio (2014), we established a patient–patient similarity network (PPSN) by using the network data visualization software Cytoscape v2.8.3 (Institute of Systems Biology, Seattle, WA, USA) (Smoot et al., 2011).

Model integration of derivative information of patients

Construction of the ANN I model and the calculation of Euclidean distance of CEIs between patients was used to obtain derivative information for each patient. For example, four items of derivative information could be obtained for patient i as follows: first, we divided the other patients except patient i into m mutually exclusive divisions of nearly equal size according to the magnitude of their distances to patient i in the n-dimensional space of ILD-associated CEIs. Thus, the four items of derivative information might be the ANN I output of patient i, mean ANN I output of patients in a given division, mean Euclidean distance to the patients in the division, and actual proportion of RA-ILD among patients in the division. The four derivative information items were imported in a new RBF-ANN as inputs to predict patients’ potential PF risk, in the same way as the ANN I. In order to distinguish it from ANN I, the new ANN was named as ANN II. We further investigated the effect of different patient groupings on the performance of ANN II. In this study, the division size m was assigned a value of 5, 10, 15 or 20. For example, if we divided patients into five divisions, we could obtain five candidates of ANN II. The Youden index calculation was then used to identify the best model as ANN II.

Model validation and performance evaluation

For ANN II, the 10-fold cross-validation method was used for model validation as previously described (Li et al., 2015). Briefly, all the patients were randomly divided into 10 mutually-exclusive sets of nearly equal size. Next, nine were selected for model training and one was used for model validation. The above procedure was repeated 10 times to allow each of the 10 patient sets to be independently used for validation.

To investigate whether the ANN models I and II identify patients with high PF risk, we performed ROC curve analysis on ILD-associated CEIs and outputs of ANN I and ANN II using MedCalc v15.8. Besides AUC we also recorded the values of sensitivity, specificity, Youden index, and calculated diagnostic odds ratio (DOR) at the optimal cut-off point. The Youden index was the sum of sensitivity and specificity minus 1 as defined previously (Youden, 1950). According to the definition of DOR in previous studies (Böhning, Holling & Patilea, 2011; Glas et al., 2003), it was calculated as follows: ${\displaystyle \mathrm{DOR}=\frac{\mathrm{sensitivity}\times \mathrm{specificity}}{\left(1-\mathrm{sensitivity}\right)\times \left(1-\mathrm{specificity}\right)}.}$ In addition, a set of 87 RA patients suffering from PF (the second affiliated hospital of HMU) was used for independent evaluation of each ILD-associated CEI and ANN models I and II. A further set consisting of 72 RA patients with healthy lungs and 146 RA patients suffering ILD (the first affiliated hospital of HMU) was used for external validation of ANN model II. The true positive rate (TPR) was calculated when the optimal cut-off point in the ROC curve was used as the discriminant threshold. TPR was calculated as follows: ${\displaystyle \mathrm{TPR}=\frac{\mathrm{TP}}{\mathrm{TP\; +\; FN}}\times 100\text{\%}}$ where TP and FN are abbreviations of true positive and false negative.

Statistical analysis

MedCalc v15.8 was used to perform pairwise comparisons of the ROC curves based on the methodology of DeLong, DeLong & Clarke-Pearson (1988). Differences were considered as statistically significant when P < 0.05.

Patients

A total of 838 patients with RA were included in this study. Among them, 620 patients from the second affiliated hospital of HMU were subjected to complete clinical examination (Tables S1 and S2). HRCT examination verified the complication of PF in 87 of the 620 patients. In addition, 169 patients were identified as having healthy lungs and 364 were diagnosed as complicated with ILD. For 218 patients with RA from the first affiliated hospital of HMU, only four CEIs were retrieved from the hospital’s electronic system of medical records (Table S3), among which 72 were identified as having healthy lungs and 146 were diagnosed as complicated with ILD.

Identification of ILD-associated CEIs

For each of the 32 CEIs (Table S1), we investigated any potential association with the status of ILD. Compared with patients with healthy lungs, those RA patients suffering from ILD have an implied greater PF risk (Brown, 2007; Ingegnoli et al., 2012). Eight CEIs were identified as ILD-associated CEIs (Table 1). For an ILD-associated CEI, higher Youden index suggested better effectiveness as diagnostic marker of PF risk (Table 1). For instance, among the 32 CEIs, age was assigned the highest Youden index or 0.301 (Fig. 1A). This characteristic was highlighted in the pairwise comparisons of the ROC curves as compared with EO, PLT or WBC (the methodology of DeLong et al., P < 0.001, Fig. 1B). The Youden index calculation indicated that age and three blood CEIs (EO, PLT, and WBC) were assigned the highest Youden indices, implying relatively closer association with ILD status (Fig. 1A). Consistent with this, the optimal ANN model effect was obtained by using the four CEIs (age, EO, PLT, and WBC) as joint inputs of ANN I (Fig. S1).

Euclidean distance calculation for networking patients’ similarity

By refining the four ILD-associated CEIs (age, EO, PLT, and WBC) as coordinates in a four-dimensional space, we calculated the Euclidean distance between any two patients and mapped a PPSN for the 533 RA patients including 169 with healthy lungs and 364 complicated with ILD (Fig. 2). In spite of the application of different edge settings, a huge patient cluster, rather than multiple scattered patient clusters, was always observed in the network.

Application of ANN for integration of ILD-associated CEIs

An RBF-ANN with 4-12-1 architecture, named ANN I, was constructed for integration of the four CEIs that were related to PF status, namely age, EO, PLT, and WBC. Compared with single CEIs, the outputs of ANN I had a better ability to identify patients with high PF risk (Youden index =0.387, Table 2). Furthermore, we built a series of RBF-ANNs with 4-12-1 architecture by calculating Euclidean distance among patients, distributing patients into divisions, and generating derivative indicators (see ‘Methods’). Regardless of the size of the divisions, the ANN generated by the division containing most similar patients showed the best effect in identifying potential PF risks (Fig. 3A). Compared with smaller grouping size, the five division method led to creation of an optimal ANN model, namely ANN II (Youden index =0.436, Table 2). A significant difference was observed in AUC when ANNs I and II were compared ( P = 0.025, Fig. 3B), suggesting the advantage of ANN II in identifying potential PF risk. The DOR of ANN I was 5.41 and that of ANN II was 6.88. When specificity was fixed as 0.80, the sensitivity of ANN I was 0.473 while that of ANN II was 0.591. These assessments indicated that ANN II was more effective and sensitive than ANN I in identifying patients with high PF risk.

Evaluation of the models

The holdout cross-validation method was used for preliminary validation of the ANNs established in this study. Two model generalization ability parameters R_Tr and R_Te were calculated using SNN software. Similar values of R_Tr and R_Te indicated that the model built using the IPS tool had good generalization ability. For ANN I, R_Tr and R_Te were 0.398 and 0.440, respectively, while for ANN II, R_Tr and R_Te were 0.461 and 0.489, respectively. For ANN II in particular, the 10-fold cross-validation method was applied for further model validation (Fig. 3C). The AUC was 0.751, implying effectiveness of ANN II in identifying potential PF risk among the patients. Furthermore, an independent set of 87 RA patients complicated with established PF was used to explore whether patients with PF could be identified by single ILD-associated CEIs, ANN I, or ANN II. Compared with single ILD-associated CEIs, ANNs I and II had better recognition for these patients (TPR = 82.8%, Fig. 4A). Consistent with this, a dot column chart visually showed that patients with high PF risk and those that had been complicated with PF could be successfully identified with higher sensitivity and specificity using ANN II (Fig. 4B). A validation set of patients from the first affiliated hospital of HMU was used for further assessing the model effect of ANN II (Table S3, Fig. 4C). The AUC was 0.792, implying the effectiveness of ANN II to identify ILD A TPR of 84.9% was calculated for identifying ILD by drawing a 2 × 2 contingency table of the validation set (Table S4).