Predicting the risk of acute respiratory failure among asthma patients—the A2-BEST2 risk score: a retrospective study

Study design and population

The modeling group comprised BA patients hospitalized at the Second Affiliated Hospital and Yuying Children’s Hospital of Wenzhou Medical University between 2017 and 2022. The validation group consisted of BA patients hospitalized at the First Affiliated Hospital of Wenzhou Medical University between 2019 and 2022. These are China’s two large tertiary hospital. Our target population consisted of patients over 18 years of age who were admitted for BA but did not have ARF at admission. According to the diagnostic criteria established by The Global Initiative for Asthma (GINA), the following inclusion criteria were used for the study: (1) All cases meetevery case met the diagnostic criteria for asthma; (2) clinical symptoms and characteristics of asthma were present; (3) medical history: recurrent wheezing, dyspnea, chest tightness or cough, primarily related to exposure to allergens, viral upper respiratory tract infection, physical and chemical stimulation, cold air, and exercise; (4) signs: During an attack, scattered or diffuse wheezing is audible in both lungs, primarily during the expiratory phase, and the expiratory phase was prolonged; (5) reversible: The above symptoms can be relieved by treatment or themselves. Exclusion criteria were as follows: (1) severe heart, liver, kidney, and other organ diseases; (2) upper airway obstruction (lung cancer, BTB, relapsing polychondritis, and tracheal foreign body); (3) allergic bronchopulmonary aspergillosis. The primary outcome was ARF during hospitalization. ARF was defined as breathing air with PaO₂ (Pressure of Oxygen) <60 mm Hg (one mm Hg = 0.133 kPa) and/or PaCO₂ (partial pressure of carbon dioxide in artery) >50 mm Hg. Only the first admission data was included when patients were admitted multiple times. Informed consent was exempted due to the retrospective format of this study. This study was approved by the Ethics Committee of the Second Affiliated Hospital and Yuying Children’s Hospital of Wenzhou Medical University (Ethical Application Ref: 2021-K-148-01).

Data collection

The following data were collected retrospectively for the study: (1) demographic information including age, male sex, body mass index (BMI), economic condition (poor: annual household income: RMB0-80, 000; middle: annual household income: RMB80,000-300,000; rich: annual household income: >RMB300, 000), education level (low level: elementary school and junior high school; middle level: high school, technical or vocational school; high level: university or college), smoking, smoking time, and smoking quitting time; (2) comorbidities including bronchiectasis, chronic obstructive pulmonary disease (COPD), emphysema, destroyed lung, thoracic deformity, cardiac insufficiency, valvular heart disease, cardiac enlargement, and area of pulmonary infection; (3) laboratory tests include albumin, prealbumin, C-reactive protein (CRP), white blood cell (WBC), and procalcitonin (PCT), (4) vital signs include temperature, heart rate, breath, mean arterial pressure (MAP), and oxygen saturation (SpO₂); (5) treatment includes onset-to-treatment time, hormone initiation time, the route of hormone administration, and long-term regular medication treatment (Table 1). The data on laboratory results were collected during the first examination before ARF (within 24 h of admission).

Model development

Three models were developed in different ways: a statistics-driven model (model A), a clinical knowledge-driven model (model B), and a decision tree model (model C) to ensure the model’s stability and validity. The three models’ performance was compared.

Model A: the statistics-driven model. The model utilized statistical analysis to determine the model’s predictive variables. The least absolute shrinkage and selection operator (LASSO) method and multivariate logistic regression analysis were applied to select variables. LASSO regression is a variable selection method with high model stability. By using the penalty factor (λ) in the model, the regression coefficients are continuously compressed, and parts of them are compressed to zero to exclude these variables from the model. LASSO can effectively deal with multicollinearity and overfitting (Jatuworapruk et al., 2020). In the regression coefficient plot, each curve represents the change track of each independent variable coefficient (Fig. S1). LASSO regression was utilized to screen 12 variables. These variables were included in the multivariable logistic regression model. The variables with P < 0.05 were eliminated. Continuous variables were transformed into categorical variables based on quartiles and clinical reality. Then, multivariate logistic regression was performed on all categorical variables to determine the final predictor variables in the predictive scoring system. To correct for over-optimism, we further shrunk the regression coefficient by multiplication with a linear shrinkage factor. The shrinkage factor was calculated using the following formula: sf = [model χ² − (df − 1)]/model χ². Model χ² represented the χ² value of the model, whereas df represented a degree of freedom (Van Houwelingen & Le Cessie, 1990). The prediction model was constructed based on the adjusted β regression coefficient. β regression coefficient represents the amount of change in the tested variable when the covariate is increased by one unit (Razzaghi et al., 2013). This model was developed by allocating an integer or half an integer score, assigning a value of 1 to the smallest regression coefficient, calculated by dividing the regression coefficient of each predictor variable by the smallest regression coefficient.

Model B: clinical knowledge-driven model. All variables were included in univariate logistic regression. The variables were preselected based on clinical knowledge and included in multivariate logistic regression. Seven significant variables obtained from multivariate logistic regression analysis were included in the model. Then, these continuous variables were transformed into categorical variables, and multivariate logistic regression was performed to build a model based on the β regression coefficients of the variables.

Model C: decision tree model. The decision tree algorithm is a machine learning algorithm. The C5.0 algorithm was used to construct the decision tree classifier, and the classifier was used to generate the intelligent tree model and the rules of the tree structure. The C5.0 decision tree algorithm allowed no assumptions regarding the statistical distribution of data, it can utilize non-parametric input data, and its application in the ARF prediction model was considered a reasonable choice for this study. The decision tree C5.0 algorithm used the information gain rate as the optimal segmentation criterion for candidate variables. The next variable was split according to the C5.0 algorithm to create a tree structure. The C5.0 algorithm used the post-pruning method to prune upward from leaf nodes, and the key issues were the error estimation and the setting of pruning criteria. Pruning was determined using the “error reduction” method. Initially, the weighted error of leaf nodes in the subtree to be pruned was calculated. Then, the error was compared to the error of the parent node. If the former error was greater than the latter, it could be subtracted. The result was represented by a decision tree, with each node depicting a different class of variables.

Model performance comparison and model validation

The model performance was evaluated using discrimination, calibration, and precision. Discrimination refers to the ability to distinguish whether an outcome occurred or not. The area under the receiver operating characteristic curve (AUC) was used to examine the model’s discrimination. AUC ≥ 0.8 indicated excellent discrimination, AUC ≥ 0.7 indicated acceptable discrimination, and AUC < 0.7 indicated poor discrimination (Weng et al., 2021). The AUC is summarized with the concordance statistic (C-statistic). The Hosmer-Lemeshow (H-L) goodness-of-fit test was used to verify the calibration degree of the model. The Hosmer–Lemeshow goodness-of-fit test showed p > 0.05, indicating no significant difference between predicted and observed probabilities. The overall performance of the prediction model was quantified as the Brier score, representing the mean squared difference between actual and predicted ARF, including discrimination and calibration. Brier scores ranged between 0 and 1, with lower scores indicating better model performance.

Model validation

The bootstrap resampling method was used to validate, with 1,000 bootstrap samples internally. A bootstrap resampling sample with the same sample size is constructed by putting back sampling in the model development queue. The process is repeated 1000 times by using this sample as a training set and the model development queue as a verification set to evaluate the performance of the model. An ideal prediction model would have similar performance in the development and validation datasets (Jatuworapruk et al., 2020). Another dataset was used for the external validation of the predictive performance. Statistical analysis was performed using IBM SPSS Statistics software (V.26; SPSS Inc., Chicago, IL, USA) and R (version 3.6.1; R Core Team, 2019).

Cohort

The study flow diagram was shown in Fig. 1. This study included 398 patients in the final analysis, including 298 patients in the modeling group, 88 patients developed ARF, and 100 patients in the validation group, 26 patients developed ARF. The modeling group had a 30% ARF incidence, while the validation group had a 26% ARF incidence. Table 2 presents the characteristics of the cohort.

Prediction models

We used three methods to generate three predictive models with different variables to predict ARF occurrence in BA patients during hospitalization. Three variables were consistently selected across all three models: economic condition, BMI, and albumin (Table 3). In model A, LASSO regression preselected twelve candidate variables. Figure S1 shows the results of LASSO regression. Finally, seven variables significantly associated with ARF were selected by multivariate logistic regression. Twelve predictive variables were retained based on clinical knowledge in the clinical knowledge-driven model (model B). Multivariate regression analysis confirmed the association of seven variables with ARF. Table 4 display the regression coefficients of each variable in model B. Table S1 provides the regression coefficients of each variable in model A. The decision tree produced eleven variables, and Fig. S2 described the relationship between the eleven predictive variables and ARF.

Models A and B have C-statistics of 0.856 (0.810−0.902) and 0.862 (0.820−0.904), respectively (Fig. 2, Table 3). Model B has a higher discrimination degree and is convenient. The C-statistics of model C was 0.846 (0.821−0.872), and the model contained eleven variables, and the practicability was poor. The Hosmer-Lemeshow test revealed that model A and model B had good calibration (Table 3). The Birer scores of models A, B, and C were 0.1331, 0.1320, and 0.1350, respectively (Table 3). According to the Hosmer-Lemeshow test, model A and model B had good calibration in external verification, and the three models’ Brier scores were 0.0645, 0.0640, and 0.0773, respectively (Table S2). Brier scores were similar for the three models. In external verification, model A and B also had similar C-statistics of 0.897 (0.822−0.971) and 0.890 (0.815−0.965), respectively (Table S2, Fig. 2). According to the clinical practice, the model’s performance, and the clinical significance of each variable, we finally choose the clinical knowledge-driven model (model B) (Fig. 3, Table 3).

Finally, we performed bootstrap validation with 1000 samples for the selected model B. The optimism corrected C-statistics was 0.854 (0.820−0.900) (Table 3). The clinical knowledge model retained good discrimination and calibration in the 1,000-time bootstrapping test set.

The A₂-BEST₂ risk score

According to a clinical knowledge-driven model (model B), seven predictive variables were used to develop the risk score for ARF occurrence in BA patients during hospitalization: the A₂-BEST₂ risk score (A₂ (area of pulmonary infection, albumin), BMI, Economic status, Smoking, and T₂ (hormone initiation time and long-term regular medication treatment)). Each variable was scored according to its regression coefficient (Table 4). The risk score for ARF in each BA patient was calculated according to the score corresponding to each variable. The A₂-BEST₂ risk score ranged from 0 to 13.5. BA patients were divided into four groups based on their scores: low-risk (0–4 points), moderate-risk (4.5–8 points), high-risk (8.5–11 points), and very high-risk (11.5–13.5) groups (Table 5, Fig. 3). In the four groups, the predicted probabilities of ARF were 1.37%, 14.92%, 57.40% and 90.32%, respectively. In different risk groups, the actual probability of occurrence was close to the predicted probability. The risk stratification of the score placed 44 (14.8%) cases in the low-risk group, 153 (51.3%) cases in the moderate-risk group, 81 (27.2%) cases in the high-risk group, and 20 (6.7%) cases in the high-risk group. Furthermore, the ARF probability predicted by risk score was very close to that of actual ARF probability in the different risk groups (Table 5). We compared the predictive power of the prediction model with Sofa score and MEWS. The area under the ROC curve of Sofa score and MEWS for predicting ARF in BA patients was 0.668 (0.598−0.737) and 0.628 (0.555−0.701), respectively (Fig. S3). The A2-BEST2 Risk Score predictive capability was superior to the Sofa score and MEWS.