Novel statistically equivalent signature-based hybrid feature selection and ensemble deep learning LSTM and GRU for chronic kidney disease classification

SES feature selection

In the proposed work, the SES method is considered to identify the important variable in CKD. The SES method is a constraint-based feature selection technique grounded in causal analysis theory. This approach identifies the optimal set of predictors for a target variable, known as the Markov blanket (MB), within a Bayesian network (BN) under specific assumptions. Bayesian networks offer compact representations of multivariate distributions using a direct acyclic graph (DAG) and precise parameterization, where nodes represent random variables and edges signify conditional relationships. In conditional relationships, if an edge connects two nodes, their variables exhibit association among all other variables. The SES method, as hypothetically implemented here using the Gini Index, assesses the significance of each feature by examining its statistical impact on the model. Traditionally, the Gini Index is employed in decision trees to quantify node impurity, but in this scenario, it is utilized to gauge the relevance of individual attributes. Features are then ranked based on their importance scores, with those exceeding a specified threshold identified as significant.

In Algorithm 1 , presented in pseudocode, SES is outlined. This method takes as input a dataset D_set, a target variable Tar, and two hyperparameters: a threshold for discerning conditional independence thres and an upper limit max on variables within a conditional set. These parameters constrain the algorithm’s computational complexity and resource requirements. The procedure generates a set K of variables, organized into queues Q with i = 1 to N and each queue containing equivalent variables.

During initialization, an empty set Set_s of selected variables is created, encompassing all variables v and added in Set_s, and L←v represents the list of variables, and each variable is equivalent only to Q_i←i. The algorithm runs a loop, and the aim is to include the variable most correlated with Tar based on any subset of selected variables and exclude any variable v not correlated with Tar given any subset sub_set of other variables in Set_s. Excluded variables v are ineligible for addition to Set_s.

Before eliminating v from Set_s, the method searches for a variable p in sub_set that matches v by validating when $sub\text{\_}set\leftarrow sub\text{\_}set\cup \left\{v\right\}\setminus \left\{p\right\}$.

LASSO feature selection

LASSO, a linear model utilizing L1 regularization, reduces the coefficients of less important features to zero, thereby conducting feature selection. The significance of each feature is reflected by the absolute value of its coefficient. Features with non-zero coefficients are deemed important and are included in the final model. LASSO feature selection enhances model interpretation and prediction accuracy. It uses ridge regression and subset selection simultaneously and selects one feature correlation while minimizing the others to zero. All regression methods aim to find coefficients for existing features, ensuring that applying these coefficients to each sample of data yields the defined outcome. The LASSO approach truncates coefficients to zero by changing them by a constant value of λ. Reduces residual sum of squares, resulting in a reduced absolute value of coefficients. The following is a definition of LASSO in its formal form, which is defined in the Eq. (1). The LASSO formula aims to discover the optimal coefficients β that balance model complexity and predictive accuracy by minimizing the sum of the L1 regularization penalty and the squared loss term. (1)${\displaystyle \widehat{\beta }=ar{g}_{\beta }min\sum _{i=1}^N{\left(y_i-{\beta }_0\sum _{j=1}^p{v_i}_j{\beta }_j\right)}^2}$

 
_______________________ 
Algorithm 1 Hybrid SES feature selection algorithm_________________________________ 
  1:  Input CKD Datasets. 
  2:  D_set represents datasets 
  3:  v independent variables 
  4:  Tar represents dataset target variables 
  5:  thres represents threshold 
  6:  A collection of data on n independent predicting variables i. 
  7:  The variable t is the target. 
  8:  Indicator of threshold th variable 
  9:  Max set is representing variable max 
10:  A set max of N variables Qi = 1toN can be used to generate a signature 
     by selecting only one variable from each set. 
11:  List of variables L ← v 
12:  Variables selected. 
13:  Set _s ← Φ 
14:  Equivalence sets 
15:  Qi ← i,fori=1toN 
16:  While L ⁄= Φ do 
17:  for all vϵ{L ∪ Set_s} do 
18:  L ← L ∖{v} 
19:  Set _s ← Set_s ∖{v} 
20:  sub_set ← sub_set ∪{v}∖{p}. 
21:  Remove irrelevant feature. 
22:  Input CKD features to LASSO feature selection method. 
23:  ^ β = argβmin∑N 
    i=1 ( 
     yi − β0 ∑p 
     j=1 vijβj) 
                 2 
24:  List the selected features from SES and LASSO____________________________________    

Ensemble deep learning with LSTM and GRU

In phase 2, we proposed the ensemble-based deep learning approach to classify CKD. Most of the work considered individual classifiers to classify the task. Moreover, CKD datasets are nonlinear. Using individual classifiers to classify the datasets may not produce good accuracy because of nonlinearity in the data. Therefore, this work considered the ensemble-based deep learning method to address the problems.

In Fig. 2, the proposed ensemble-learning architecture is depicted. This architecture involves training two separate deep-learning models using the same dataset and then merging their predictions to produce the final prediction result. The approach combines two distinct classifiers, with each model dedicated to a specific binary class output. We selected these two classification models with the intention that one would outperform the other in one of the output classes. This selection was based on pilot tests demonstrating the superior performance of proposed LSTM and GRU models for the majority output class 1, corresponding to chronic kidney disease. Class 0, indicating the absence of chronic kidney disease, represented the minority output. (2)${\displaystyle f_t=\delta \left(w_f.\left[{h_t}_{-}1,v_t\right]+b_f\right)}$ (3)${\displaystyle i_t=\delta \left(w_i.\left[{h_t}_{-}1,v_t\right]+b_i\right)}$ (4)${\displaystyle C_t\tilde{}=tanh\left(w_C.\left[{h_t}_{-}1,v_t\right]+b_C\right)}$ (5)${\displaystyle C_t=f_t\ast C_t-1+i_t\ast C_t\tilde{}}$ (6)${\displaystyle i_o=\delta \left(w_o.\left[{h_t}_{-}1,v_t\right]+b_o\right)}$ (7)${\displaystyle h_t=o_t\ast tanh\left(C_t\right)}$

LSTM neural networks were developed using a series of interconnected neural network modules to address the problem of long-term dependency. An LSTM-based model is constructed to depict the development of CKD because of LSTM’s ability to understand long-term dependencies. To forecast CKD, we have considered three layers: the pre-fully connected layer, the cells layer, and the post-fully connected layer, as described in Fig. 2. The first fully connected layer has a ReLU function and one fully connected layer; the Second Layer has one LSTM layer and a dropout wrapper; and the third layer has one completely connected layer and a softmax layer.

In an LSTM network, there are four main components: the “forget gate”, “input gate”, “update gate”, and “output gate”. The “forget gate” Eq. (2) determines whether data should be removed from the cell state. The “input gate”, which consists of sigmoid and tanh layers, decides which values should be updated, and it defines Eqs. (3) and (4). The “update gate” in Eq. (5) takes the value from the “input gate” and uses it to update the previous cell state. Finally, the “output gate” is defined in Eqs. (6) and (7) determines the value to be output from the layer. f_t is a number between 0 and 1, where 0 signifies forget and 1 signifies keep; W_f is the weight matrix, and b_f is the bias vector. f_t determines which data to be forgotten and $\widetilde{C_t}$ determines the number of cell values to be changed. The i_o value in Eq. (7) determines which component of the cell state provides the output. Equation (7) yields h_t, the output of the components i_o, by multiplying the new cell state $\widetilde{C_t}$ by o_t and the function tanh that has been chosen. (8)${\displaystyle r_t=\delta \left(w_i^T\left(v_t\right)+w_i^{T}o\left(t-1\right)+b_r\right)}$ (9)${\displaystyle z_t=\delta \left(w_i^T\left(v_t\right)+w_i^{T}o\left(t-1\right)+b_z\right)}$ (10)${\displaystyle o_t=z_t\otimes o\left(t-1\right)+\left(1-z_t\right)\otimes o_t}$

Gated recurrent unit (GRU)

In contrast to LSTM, GRU uses fewer gates in its RNN architecture. In a GRU cell unit, a single gate controls both the input and forget functions. To simplify, GRU uses a single gate for both inputs and forgets, making it less complicated than LSTM, and its equations are defined in Eqs. (8), (9) and (10). For example, when zt = 1, the mechanism opens the forget gate and closes the new data entry for the input gate. The new state is computed by combining the new input with the previous memory, which the reset gate decides. Where r_t is the reset gate, z_t is the update gate, and o_t is the output gate. ⊗ denotes element-wise multiplication; t is the time step. The window length is represented by T. The layer weight w reflects input v, while b represents the output gate threshold.

Support vector machine

An SVM learning system uses a high-dimensional feature space. SVM classifies points by distributing them to one of two separated half spaces: the pattern space or a higher-dimensional feature space. The primary goal of SVM is to find the largest margin hyperplane. The objective is to maximize the difference between positive and negative classes.

The maximum margin hyperplane is the final decision boundary. Consider that v_i ϵ K^d, i = 1 to N forms creates input vectors with class labels y_i $ϵ\left\{CKD-Yes,CKD-No\right\}$. SVM maps input vectors v_i ϵ K^d to a high-dimensional feature space Φ(v_i)ϵ Hyperplane. Kernel function kernal(v_i, v_j) maps ϕ(.). The decision boundary is defined in the Eqs. (11), (12) and (13). (11)${\displaystyle f\left(v\right)=sgn\left(\sum _{i=1}^N{y}_i.{\alpha }_i.Kernel\left(v,v_i\right)+b\right)}$ (12)${\displaystyle Max\sum _{i=1}^N{\alpha }_i-\frac{1}{2}\sum _{i=1}^N\sum _{i=1}^N{\alpha }_i{\alpha }_j.y_i{y}_j.Kernel\left(v_i,v_j\right)}$

subject to (13)${\displaystyle 0⩽{\alpha }_i⩽c}$

Logistic regression

Logistic regression is a popular statistical analysis technique that uses predictive factors to determine the probability of an outcome. LR considered a convergence criterion to maximize likelihood functions in Eq. (14).

In this model, prob represents the probability of the result for the expected predictive variable, k0 is the intercept condition, k1, k2, …, kn are regression coefficients, and v1, v2, …, vn are prospective predictive variables. (14)${\displaystyle y=\left(fprob\right)=log\left(\frac{prob}{1-prob}\right)={\alpha }_0+{\alpha }_1{v}_1+{\alpha }_2{v}_2+{\alpha }_n{v}_n.}$

Decision tree

The decision tree (DT) is a machine-learning classification method based on tree structures. DT is designed to create a binary classification tree with root, internal, and leaf nodes. The root node contains input data; internal nodes represent decision function branches and leaf nodes display the output.

Training a decision tree involves two main phases: (1) constructing the classification tree and (2) pruning the tree. In the first phase, we start at the root of the classification tree and identify the input data with the highest gain ratio. Sub-nodes are then generated based on the splitting method applied to the training dataset. The second phase involves assigning a unique gain ratio value to each sub-node and using these values to establish the classification variables.

Random Forest classifier

The Random Forest (RF) algorithm is an ensemble learning method based on decision tree learning. Ensemble learning suggests that a single classifier may not accurately classify test data because it cannot effectively differentiate between noise and patterns based on sample data. The random forest classifier trains several decision trees (n trees) using sampling with replacement from the given data sets. Each tree is trained using a random selection of three attributes. After evaluating the data, the final output is determined by the majority decision of the n trees.