Applying stacking ensemble method to predict chronic kidney disease progression in Chinese population based on laboratory information system: a retrospective study

Introduction

Chronic kidney disease (CKD) is a worldwide clinical and public health problem that is associated with major adverse health events, including kidney failure, cardiovascular disease, and mortality. It leads to poor quality of life and high costs of treatment (Luyckx et al., 2023). In China, CKD is integrated into the national public health surveillance program. The subgroups with a greater incidence of CKD included older age, female, sex, non-Han ethnic background, and living in rural areas or in the northern and central regions of China (Wang et al., 2023). Over the past 20 years, the number of CKD-related deaths has increased by 82.3% worldwide, especially for CKD stages 4 and 5 (Bello et al., 2017). Early identification of patients who may develop renal dysfunction in the future and proper management of CKD and its risk factors are crucial for slowing the progression of the disease. It can effectively reduce progression to end-stage renal disease (ESRD) and cardiovascular mortality and reduce the economic burden of CKD.

Currently, several clinically reliable biomarkers are available to predict progressive CKD. Traditional kidney biomarkers, such as serum creatinine and proteinuria, have many well-recognized limitations, including a lack of specificity and sensitivity in predicting a decline in renal function and delayed detection of kidney injury. Studies employing new technologies have focused on identifying structural markers of kidney tubular and glomerular injury in urine or blood, but these biomarkers are not available or affordable in clinical practice (Owens et al., 2022). Multiple risk factors for CKD have been identified, including older age, diabetes mellitus, hypertension, proteinuria, hyperuricemia and hyperphosphatemia (Kawasoe et al., 2023; Samanta, Bandyopadhyay & Samanta, 2023). Establishing renal prognostic prediction models based on the evaluations of these variables is a hot and difficult issue in current research. It may be helpful and convenient to identify those at high risk who may benefit from more intensive management, reduce progression to ESKD and cardiovascular mortality, and, eventually, improve health system efficiency. Several studies have developed models for predicting worse clinical outcomes in CKD patients. Tangri et al. (2011) developed kidney failure risk equations (KFREs) that can accurately predict the progression to kidney failure. Kawasoe et al. (2023) performed multivariable logistic regression analysis and assigned scores to each clinical factor to predict CKD. However, this equation differs in the target population’s ethnicity and CKD risk factors, and the factors associated with CKD development vary by race (Harada et al., 2022; Nelson et al., 2019). We need a unique Chinese predictive model for CKD progression to identify high-risk patients in the Chinese population. Additionally, CKD is generally managed by primary care physicians in China. The development of convenient and clinically useful tools for identifying high-risk CKD patients and providing appropriate counseling where clinical resources are limited is vital.

More recently, machine learning (ML) approaches have shown excellent ability to identify variables relevant to clinical outcomes, predict better performance, model complex relationships better, and overcome the limitations of expert systems. A strong growth of diagnostic equations for which ML algorithms are designed. Xiao et al. (2019) developed diagnostic models by deep neural networks to quickly predict the severity of CKD based on more easily available demographic and blood biochemical features instead of urinary protein and showed good predictive ability. For the prediction and risk stratification of kidney outcomes, Chen et al. (2019) used the extreme gradient boosting (XGBoost) algorithm to select the 10 most important variables and established a prediction model that can stratify the risk for kidney disease progression in the setting of IgAN. However, the risk stratification system is specific to IgA and cannot be generalized or applied to other types of kidney disease (Chen et al., 2019). Ferguson et al. (2022) reported that random forest algorithms can predict estimated glomerular filtration rate (eGFR) decline or kidney failure accurately. Nevertheless, the model involves 22 variables; from a clinical point of view, although a complex scoring system may provide accurate predictions, it is inconvenient in daily clinical practice. Furthermore, it was targeted at CKD stages G1 to G5, which is generally too wide, and it is too early to plan possible dialysis sessions since the G1 stage (Ferguson et al., 2022). Recently, researchers have focused more on the performance of ensemble learning in medicine than on the performance of a single ML algorithm (Li et al., 2024). Stacking is a powerful method for integrating different types of base learners and is rarely used in CKD prediction.

In this study, we developed an accurate but simple online prediction model based on stacking methods using a new dataset from the Chinese CKD population with long-term follow-up and evaluated its validity in an external validation set. The aim was to use variables routinely measured in patients with CKD to create an online calculator to predict progression to kidney failure that could be easily and quickly applied in clinical diagnosis and treatment. The model can be generalized to assist clinicians in managing CKD in primary hospitals in China.

Materials and Methods

Machine learning-based model development and evaluation

(1) Feature selection

Feature selection over the initial set of included variables is needed to reduce the probability of overfitting and improve the generalization capabilities of predictive models. In the training cohort, least absolute shrinkage and selection operator (LASSO) logistic regression analysis was used to select laboratory variables that best-identified high-risk patients. The LASSO regression reduces the β coefficient of variables not strongly associated with the outcome to 0 and eventually removes these variables from the model. Two different λ optimization criteria, lambda with the standard error of the minimum distance (λ-1SE) and lambda with the minimum mean square error de (λ-min), were selected in this study to construct the prediction model.

(2) Stacking ensemble technique

The stacking ensemble model is an ML technique that combines multiple base models to improve predictive performance. In stacking, meta-models are trained to learn how to best combine the predictions of the base models. The stacking models include base models and meta-models. The base-model classifier is trained using the initial training dataset. The output of the base model is used as the input feature of the meta-learner. A new dataset is then formed using the corresponding original labels as new labels to train the meta-learner. First, we use three algorithms, XGBoost, LightGBM and Random Forest, as the base models. The meta-model included a logistic regression model, which combined the predictions from the aforementioned three base-learners, analogous to the propensity score method (Li, Stein & Nallasamy, 2023). In this study, the aim of using an integrated ML model is to take full advantage of different classes of ML algorithms and improve the overall performance of the prediction model. K-fold cross-validation (k = 5) was applied to the modeling dataset, and various parameter combinations were exhausted via grid search.

Models were evaluated for accuracy via the area under the receiver operating characteristic curve (AUC) with 95% confidence, F1 score, sensitivity, specificity, positive prediction value (PPV), and negative prediction value (NPV). The calibration curve was assessed for the prediction of the outcome at 1 to 4 years in 1-year intervals, and decision curve analysis (DCA) assessed the clinical benefit of the model. SHAP is used to interpret models, which calculates the contribution and influence of each feature toward the final prediction precisely. The SHAP values describe how each predictor enhances or detracts from the outcome variable. The workflow to develop the prediction model of CKD progression is shown in Fig. 1.

Results

ML model establishment and evaluation

LASSO regularization was used to select laboratory variables that best identified patients at high risk for ESRD. Since the eGFR is used to define the outcome and is calculated from the serum creatinine level, the eGFR and creatinine level were not considered in the assessment. Based on the two different λ optimization criteria outlined in the “Methods” section (λ-1SE = 0.103, λ-min = 0.054), three and six out of 46 potential predictors yield the λ-min and λ-1SE models, respectively. The coefficient profile and the cross-validated error plot of the LASSO regression model are shown in Fig. 2. The derived laboratory variables included 24hrUpr, GLU, and urea. The second variable combination, derived from six laboratory variables, comprises potassium, prealbumin (PA), and TP, in addition to the aforementioned three laboratory variables.

In the test set, the AUC for predicting CKD progression ranged from 0.806 to 0.896 when different learning methods and combinations of laboratory variables were used. Numerically, the stacking meta-classifier logistic regression with six variables achieved the best prediction performance, with an overall mean AUC of 0.896. Figures 3 and 4 show the ROC curves of each ML model. The accuracy, sensitivity, specificity, PPV, NPV, and F1 scores for this model in cross-validation were 0.824, 0.941, 0.765, 0.789, 0.867, and 0.859, respectively (Table 2). The results of the confusion matrices are summarized in Tables S2 and S3. In addition, the DCA curve showed that the model had high clinical benefits in the approximately 30% to 70% risk range, as shown in Figs. 3C and 4C. We finally chose those six laboratory variables to construct a prediction model of CKD progression.

Table 2:

Comparative analysis of the model performance.

		ML model	AUC	Accuracy	Sensitivity	Specificity	PPV	NPV	F1 score
Three Laboratory variables	Training cohort	XGBoost	1	0.994	1	1	1	0.988	1
		LightGBM	0.97	0.918	0.916	0.933	0.933	0.906	0.924
		RandomForest	1	0.987	0.995	0.999	1	0.974	0.998
	Internal validation cohort	XGBoost	0.814	0.712	0.865	0.759	0.769	0.676	0.812
		LightGBM	0.814	0.759	0.829	0.753	0.749	0.772	0.784
		RandomForest	0.806	0.735	0.812	0.747	0.761	0.719	0.78
		Logistic (stacking)	0.862	0.794	0.882	0.765	0.778	0.813	0.827
Six Laboratory variables	Training cohort	XGBoost	1	0.993	1	1	1	0.987	1
		LightGBM	0.993	0.963	0.951	0.989	0.989	0.94	0.969
		RandomForest	1	0.983	1	1	1	0.967	1
	Internal validation cohort	XGBoost	0.847	0.712	0.847	0.788	0.865	0.651	0.855
		LightGBM	0.864	0.759	0.871	0.8	0.836	0.711	0.853
		RandomForest	0.855	0.718	0.847	0.765	0.822	0.669	0.83
		Logistic (stacking)	0.896	0.824	0.941	0.765	0.789	0.867	0.859

DOI: 10.7717/peerj.18436/table-2

Note:

AUC, area under curve; ML, machine learning; NPV, negative predictive value; PPV, positive predictive value.

SHAP values were applied to determine the impact of the risk factors to detect the positive and negative relationships of the indicators with CKD progression. The importance of the features in the ensemble models, XGBoost, LightGBM and Random Forest are shown in Fig. S1. The attributions of all patients to the results are plotted with red dots indicating that the value of the feature is larger and blue dots indicating that the value of the feature is smaller. Compared with the nonprogressive cohort, increased 24hrUpr, GLU, urea and potassium, and decreased PA and TP contributed to the prediction of the risk of renal failure. The rankings of the six features were evaluated by the average absolute SHAP value.

Performance of the model with different follow-up periods

The prediction model based on the six laboratory variables yielded high ESRD AUCs of 0.773, 0.798, 0.841, and 0.87 for years 1, 2, 3, and 4 after the baseline visit, respectively (Table 3). The calibration curve showed good model prediction in terms of Brier scores evaluated on the test sets across all time points (Fig. 5). This demonstrated that the stacking model constructed in this study can predict the CKD status of patients across different periods. The visualization of the predictive model is implemented as an online web service tool available at http://www.xsmartanalysis.com/model/list/predict/model/html?mid=10255&symbol=817bW025614Bs59zK1sJ.

Table 3:

Results of the six laboratory variables model in the internal tesing cohort for prediction of ESRD in 1, 2, 3, 4 years after the baseline visit.

Time frame, year	AUC	Sensitivity	Specificity
1	0.773	1	0.574
2	0.798	0.867	0.706
3	0.841	0.971	0.685
4	0.87	0.947	0.76

DOI: 10.7717/peerj.18436/table-3

Note:

AUC, area under curve.

Discussion

Worldwide, experience indicates that the contribution of CKD to kidney dialysis, kidney transplantation and mortality is rapidly increasing (Ruiz-Ortega et al., 2020). Therefore, identifying risk factors for progression and obtaining accurate, individualized risk estimates to assist in clinical decisions on further management are important. Several studies have developed predictive scores for CKD, but these scores vary in terms of the target population’s ethnicity and the complexity of the scores, which are determined by the number and classification of factors involved. There is a need for a distinct Chinese predictive model for CKD progression to identify high-risk patients within the Chinese population. In this retrospective cohort study, we investigated possible predictive factors associated with the progression of CKD in a cohort of the Chinese population. The proposed stacking ensemble model that combines multiple base models achieves better performance in all respects and takes full advantage of different classes of ML algorithms. Furthermore, we developed an online risk prediction model based on only six easily available predictors for the occurrence of ESRD with good accuracy and satisfactory calibration that covers most patients in areas where there are only limited healthcare resources in China. The form of an online calculator should facilitate the long-term management of chronic diseases and the selection of patients most likely to benefit from innovative strategies to halt the progression of chronic kidney disease. In actual clinical practice, it can be easily and quickly applied in decision-making for treatment and lifestyle improvement and reduces the costs related to treatment.

Previous investigators have reported risk prediction models for predicting future renal dysfunction. Owens et al. (2020) developed a model for predicting progressive CKD based on a panel of biomarkers representing the pathophysiological processes of CKD and had an accuracy of 84.3%; however, deficiencies in the study included separate small sample sizes, and the included biomarkers, such as osteopontin and tryptase, are uncommon in clinical practice. Isaza-Ruget et al. (2024) studied a cohort from Colombia and developed a machine learning-based model for predicting the need for renal replacement therapy (RRT) and disease progression in patients with stage 3–5 CKD. However, the limitations of this study include its limited characteristics and uncertain data quality, which were extracted from private health insurers (Isaza-Ruget et al., 2024). Ventrella et al. (2021) applied machine learning to assess the advancement of CKD, but as many as 27 features were included in the model, which may have a high risk for overfitting and is not convenient for clinical application. Zhu et al. (2023) constructed recurrent neural network models to predict CKD progression from stages 2–3 to stages 4–5 based on time series records extracted from the EHR. However, the model was not validated in an external validation set.

As described above, appropriate adequate risk prediction models have shown high predictive power in predicting CKD risk, but they are not easy to calculate and do not translate into routine clinical practice to assist in clinical decision-making. Therefore, online calculators or others should be considered for practical application. The kidney failure risk equations proposed by Tangri et al. (2011) have shown good performance and are used to guide patient management. The model was established based on routinely obtained laboratory tests, and simple operation steps have driven its adoption (Tangri et al., 2011). It would be valuable to apply the prior risk model. However, the risk model could not be applied in this study because it requires an albumin-to-creatinine ratio, which was not used in our study, and the eGFR was used to determine outcomes. Furthermore, most studies based on a nationwide cohort of patients and risk prediction models have been constructed in various ethnic groups in different countries. For a predictive model of the progression of CKD, few studies have been conducted in the Chinese population. Since the prevalence of CKD and its risk factors vary among ethnic groups (Hsu et al., 2021), it is preferable to use Chinese data to predict CKD risk in the Chinese population.

The online prediction ML model we built showed high predictive ability for CKD progression and good discriminative ability in predicting patients with CKD in both the internal validation and external validation cohorts. Based on the LIS and EHR systems of Peking University First Hospital, our model includes various types of kidney diseases, such as IgA, membranous nephropathy, and focal segmental glomerulosclerosis, which can be widely used in clinical practice. Ensemble learning is a technique that trains to combine predictions from the base learner and has been widely used across multiple medical fields to improve predictive performance (Mahajan et al., 2023). Several diagnosis and prediction ensemble learning techniques have already been proposed for CKD. Chhabra, Juneja & Chutani (2023) used an ensemble learning approach based on the top three best-performing classifiers in terms of cross-validation results to identify patients at risk of CKD with an accuracy of 99.5%, but this high predictive power may be due to overfitting caused by the small dataset. Napa, Tulasi & Dhamodaran (2019) proposed an ensemble learning technique for predicting the occurrence of kidney disease by analyzing various medical factors that comprise a support vector machine, decision tree, C4.5, and particle swarm optimized multilayer perceptron, attaining an accuracy of 92.76%. Hasan & Hasan (2019) developed an ensemble method-based machine learning algorithm to improve the performance of classifiers for kidney disease. Five machine learning classifiers, namely, adaptive boosting, bootstrap aggregating, extra trees, gradient boosting, and random forest classifiers, were used to design the computer-aided diagnosis system, and the classification accuracy reached 99% (Hasan & Hasan, 2019). Ganie et al. (2023) used boosting techniques based on clinical parameters to predict the risk of developing CKD in at-risk populations with 98.47% accuracy in testing sets. However, most studies have focused on the detection and prediction of the incidence of CKD. Ensemble learning is less commonly used to predict the progression of renal function in CKD patients. Lu et al. (2023) developed an integrated algorithm (LR+XGBoost) to achieve good prediction performance on the CKD dataset, with AUCs of 0.856 and 17 features included in the model, whereas proteinuria was used as a standard for CKD in this study. As an extension of this work, we used ensemble learning techniques to take advantage of different classes of the ML algorithm (XGBoost, LightGBM and Random Forest) and improve the model’s overall performance. Compared with the single classifier-based model, the stacking model performs better. The model based on only six routine laboratory parameters performed well, with AUC values of 0.896 and 0.771 in the internal and external validation sets, respectively. Our risk prediction models have important implications for clinical practice, public health policy, and mechanism research. The models can be easily integrated into LISs and EHRs to aid clinical decisions. For community hospitals and healthcare centers in China, the testing equipment is limited, and clinicians are inexperienced. In daily clinical practice, our model can help physicians predict the progression of renal function based on accessibility and objective laboratory indicators and can contribute to guiding clinical treatment and patient lifestyle. For example, lower-risk CKD patients could be managed by healthcare centers and community hospitals without excessive medical treatment or testing, whereas higher-risk CKD patients could be managed by nephrologists and receive timely intervention and correct medication guidance. This method will decrease national healthcare costs, benefiting China, a developing country with a large population. In addition, developing an application that can calculate the risk prediction equation via portable laptops or mobile devices would be more convenient and clinically useful for assessing risk. CKD is a condition that requires long-term management and monitoring. Patients can use the online model to enhance their self-management capabilities, helping them better understand the progression of the disease and develop more effective management plans, thereby improving their adherence to treatment regimens. This model can also accurately predict the progression of renal function in patients with varying follow-up periods after their initial assessment. The poor performance of the calibration curve at 1 year may be because CKD is a chronic progressive disease, and fewer patients progress to ESRD within 1 year in our study, which needs to be verified in the future with a larger population. Furthermore, it needs to be aware that the developed model could serve as an initial decision-support tool for physicians, with further clinical examination needed to make more informed decisions.

By using the LASSO regularization method, six laboratory variables were selected that best identified patients at high risk for ESRD, namely, 24hrUpr, GLU, urea, potassium, PA and TP. Based on the feature importance of the ensemble models, 24hrUpr, GLU and urea were the top three indicators. Albuminuria has been studied extensively in the context of CKD and its progression. Albuminuria is a major risk factor for renal disease, and albuminuria testing is crucial for guiding evidence-based treatments to mitigate chronic kidney disease progression (Chu et al., 2023; Levey, Grams & Inker, 2022). The 24hrUpr was included in our ML model, thus confirming its importance as an independent factor in the progression of CKD. Makino et al. (2019) used big data machine learning to predict the progression of diabetic kidney disease (DKD), and blood glucose and HbA1c were selected in the model to reveal time series patterns related to 6-month DKD aggravation. Chen, Chen & Jiang (2022) reported that DKD patients have a greater risk of developing a 50% decrease in the eGFR and kidney transplantation replacement than non-DKD patients. Our study model also identified blood glucose as a risk factor for the progression of CKD. The modeling population included in this study was 38 patients with comorbid diabetes or DKD. Several studies have shown that urea is a marker of uremic retention in CKD patients and that elevated urea concentrations induce disintegration of the gut epithelial barrier and promote microhemorrhages. Urea toxicity leads to systemic inflammation and endothelial dysfunction, thus directly contributing to disorders of tubular function and the progression of kidney disease (Hobby et al., 2019; Lau et al., 2020; Rosner et al., 2022). In summary, urea is a key factor in CKD pathophysiology, and our study emphasized that minimizing urea accumulation is clinically beneficial for slowing the progression of CKD. Patients with chronic kidney disease often have abnormal serum potassium caused by a lower GFR and the administration of angiotensin-converting enzyme inhibitors, angiotensin-receptor blockers and other demographic factors (Clase et al., 2020; Kim, Valerio & Knobloch, 2023). Previous studies have suggested that hypokalemia is related to the accelerated progression of CKD. The possible reasons could be increased renal ammonia production and impaired renal angiogenesis (Gilligan & Raphael, 2017). For PAs, studies have shown that protein-energy wasting, which manifests as low serum levels of PAs, is one of the strongest predictors of mortality in patients with CKD (Barril et al., 2022). Studies have shown that higher intake of TP was associated with lower mortality in participants with CKD (Carballo-Casla et al., 2024). The reduced TP levels in our study contributed to an increased risk of CKD progression. Our study also emphasized that paying attention to the levels of PA and TP and proper nutritional support are essential for the prognosis and quality of life of patients with CKD. Although the laboratory markers mentioned above have also previously been associated with the progression of CKD and individual indicators inadequately explain the variability in the decrease in the glomerular filtration rate, our work integrates them into an online risk equation to predict the progress of kidney function and assist in clinical diagnosis and treatment.

The present study has several limitations. First, the retrospective study was restricted to Chinese patients from a single center with a relatively small sample size. Further studies including larger and multiple centers, are needed to improve and validate the risk prediction models. Second, model development was based on a retrospective cohort, which reduced the availability of complete data related to clinical characteristics, treatment-related information and laboratory indicators of interest for the final analysis. Therefore, prospective research based on full clinical information is needed to identify a better method for predicting CKD progression. Third, although we applied PSM designs to construct a balanced cohort, there may be potential bias in unmeasured or unknown factors. Fourth, the outcome of CKD progression can be subdivided, and monitoring the dynamic changes in clinical indicators during follow-up can be considered to improve the accuracy and timeliness of predicting the risk of kidney failure.

Conclusions

Overall, we applied the ML method to present an online portable model for accurately predicting the occurrence of ESRD in patients with CKD stages 3 to 5. The predictive ability of the model was confirmed through external validation. This model also accurately predicted the progression of renal function in patients with varying follow-up periods after their initial assessment. Only six easily accessible laboratory tests, which are easy to popularize in primary hospitals in China, were included in the model. The form of online calculators could be an easy and quick tool for detecting and recognizing high-risk CKD patients and for providing appropriate counseling and treatment. Moreover, the laboratory indices screened by the ML model provide a new idea and reference for managing CKD.

Supplemental Information