An ensemble learning-based feature selection algorithm for identification of biomarkers of renal cell carcinoma

Data preprocessing

Feature selector based on F-test

An F-test is a statistical method based on analysis of variance that compares variance differences between two or more samples. In the context of feature selection, the F-test is employed to assess the importance of each feature by comparing the variance differences between the response variable and each feature.

The steps involved in conducting an F-test are as follows:

1. Calculate the variance (SS) and mean (MS) between each feature and the response variable.

2. Compute the variance (SST) and mean (MST) of the population.

3. Calculate the F statistic, which is the ratio of SS/MS to SST/MST.

4. Determine the critical value of the F statistic using the F distribution table.

5. If the computed F statistic exceeds the critical value, it indicates a significant variance difference between the feature and the response variable, thereby identifying the feature as important.

A higher F value or F statistic signifies a larger variance difference between the feature and the response variable, indicating greater importance of the feature in predicting the response variable.

In this study, the feature selector based on the F-test exhibits much faster feature selection times (at the millisecond level) compared to the other four feature selectors (at the second level). Therefore, it is deemed appropriate to utilize the F-test-based feature selector in the initial stage.

Parallel feature selector

We employ both a ranking-based feature selection method (using Mutual Information) and three search strategy-based feature selection methods (ReliefF, SURF, SURF* (Urbanowicz et al., 2018)) to perform parallel feature selection on the preprocessed data.

SURF*

The SURF* algorithm is an extension of the SURF algorithm, incorporating the concept of instances that are either closer or farther away from the target. In contrast to SURF, SURF* introduces a T-threshold to determine the proximity of instances. Instances within this threshold are considered close, while those outside are regarded as far away. Notably, SURF* assigns different weights to “far” and “near” cases. Specifically, the difference in eigenvalues for hits is positively weighted (+1), whereas the difference in eigenvalues for misses is negatively weighted (−1). Moreover, we summarize the neighbor selection difference among Relief, ReliefF, SURF, SURF* in Fig. 3.

Feature combiner based on Gini coefficient

The Gini Index is a measure of feature importance, derived from the Classification and Regression Tree (CART) method (Lewis, 2000; Tangirala, 2020). In the CART algorithm, feature importance is calculated based on purity enhancement. At each node, the purity boost of each feature is calculated, and the feature with the highest boost is selected for node splitting. For classification trees in CART, the GINI value is used for node splitting. A lower GINI value indicates a purer node set and reduces the probability of misclassifying selected samples in the set  (Erhu, 2019). Specifically, the Gini coefficient for a sample is defined as: (7)${\displaystyle Gini\left(D\right)=1-\sum _{k=1}^N{P}_k^2}$ where Pk is the proportion of the Kth sample category.

Let’s consider a node t, where feature j is chosen for splitting, dividing the dataset D into subsets D1 and D2. The Gini coefficient after splitting with feature j can be calculated as: (8)${\displaystyle Gini\left(D,j\right)=\frac{\left|D_1\right|}{\left|D\right|}Gini\left(D_1\right)+\frac{\left|D_2\right|}{\left|D\right|}Gini\left(D_2\right).}$

The purity boost of feature j is defined as: (9)${\displaystyle Gain\left(j\right)=Gini\left(D\right)-Gini\left(D,j\right).}$

After constructing the decision tree, the importance scores of all features are normalized to have a sum of 1 for comparison. The Gini importance score for feature j is given by: (10)${\displaystyle \text{importance}\left(j\right)=\frac{Gai\left(j\right)}{\sum _{j=1}^{m}Gain\left(j\right)}.}$

It is important to note that for handling continuous values, the CART and C4.5 algorithms both discretize continuous features. For multi-classification problems, the CART classification tree adopts the idea of continuously dichotomizing discrete features (Bo, 2018).

Algorithm 1 provides the complete steps of the feature combinator, which can be divided into the following three steps:

First, add the selected features from subsets S1, S2, S3, and S4 of size k to the feature subset F for final selection. The remaining unselected features form the subset D, selected by the combinator.

Second, construct a CART decision tree and calculate the Gini importance of each feature in D as an evaluation index.

Third, select the k-len(F) features with the highest Gini importance and combine them with F to form the final feature subset.

Soft voting-based classifiers

Voting is a widely used combination strategy in ensemble learning, wherein multiple learners participate in a classification problem and each model’s prediction is considered as a single ”vote”. The final prediction result is determined by the majority vote among the models. In other words, a statistical analysis is conducted on the classification results of K learners, and the class with the highest frequency is chosen as the predicted class. The voting method can be categorized into hard voting and soft voting based on the voting strategies employed. An example of the voting method is shown in Fig. 4.

(1) Hard Voting Mechanism

A “majority by majority” approach is employed to select the classification results based on the prediction of each model.

(2) Soft voting

The classification rates generated by all classifiers are averaged and selected in this study. The classifier employed in this research is a soft voting classifier, which combines several individual classifiers including decision tree, naive Bayes, support vector machine (SVM), k-nearest neighbors (KNN), and random forest. This ensemble approach effectively mitigates the errors introduced by a single classifier, thereby enhancing the overall classification performance.

Dataset

The data set used in this experiment is a high-dimensional multi-class data set, which are shown in Table 1. The COIL20 dataset is a commonly used image recognition dataset, consisting of 20 objects, each represented by 72 images taken from different angles. Each image in the dataset is converted to a grayscale image of size 128 × 128. The ORL dataset is a classic face image dataset containing 400 grayscale face images of 40 individuals, with each person having 10 face images captured in different poses. The images in this dataset have a size of 92 × 112 pixels. The warpPIE10P dataset is a face recognition dataset that comprises 4,000 images from 600 individuals, exhibiting different facial expressions, lighting conditions, occlusions, and facial poses. The images in this dataset are of size 32 × 32 pixels and have been feature extracted using PCA, resulting in 4,096 features. The Prostate_GE dataset is a gene expression dataset used to predict the tumor grade of prostate cancer (Gleason Score). Each sample in the dataset represents a specific gene expression profile in normal, precancerous, and cancerous tissues. The jaffe dataset is a facial expression database consisting of 213 grayscale images of Japanese women. Each image in the dataset has a size of 256 × 256 pixels and exhibits very uniform expressions and postures. The lung dataset is a biological dataset comprising CT images of the lungs along with corresponding annotated data. The TOX_171 dataset is a compound toxicity dataset containing 171 molecules and 12 bioactivity indicators. The Isolet dataset is specifically designed for speech recognition tasks and includes randomized English word sounds. Each instance in the dataset is represented by 617 features, including 13 linear predictive coding (LPC) coefficients and 13 Mel frequency cepstrum coefficients (MFCC) per frame, as well as fundamental frequency and energy per frame. Detailed information about these datasets can be found at https://jundongl.github.io/scikit-feature/datasets.html.

Evaluation criteria

In this experiment, we employed various feature selection methods including F-test, MI, ReliefF, SURF, SURF*, and EFS-GINI to conduct a comprehensive control study. Specifically, we selected the top 1% features from the aforementioned datasets.

Accuracy, precision, recall, F1-score (f1_score), and confusion matrix are commonly used evaluation indicators in machine learning. In our study, we set the proportion of the training set and the test set is 60:40 (Arusada, Putri & Alamsyah, 2017).

For binary classification, we present the model’s prediction confusion matrix in Table 2. In this table, TP (true positive) and TN (true negative) represent the data that was correctly predicted, while FP (false positive) and FN (false negative) represent the data that was incorrectly predicted. TP indicates the correct prediction of a positive example, TN indicates the correct prediction of a negative example, FP indicates the incorrect prediction of a positive example, and FN indicates the incorrect prediction of a negative example. The criteria for accuracy, precision, recall, and F1-score are calculated as follows: accuracy (ACC) is determined by (TP+TN)/(TP+TN+FP+FN), representing the percentage of correct predictions in the total sample; precision (P) is calculated as TP/(TP + FP), indicating the percentage of correctly predicted results in the total sample; recall (R) is TP/(TP +FN), which represents the percentage of correctly predicted results in the total sample. Since precision and recall are conflicting measures, the F1-score (F1) is introduced as 2PR/(P+R) to better evaluate the performance of the representation learner in terms of precision and recall. The closer the F1-score is to 1, the better the classification effect. For the multi-class classification, ACC and F1 can also be calculated using the confusion matrix.

Experimental environment and parameter settings

The experimental environment of the proposed methods EFS-GINI and the parameter settings in different methods are summarized in Tables 3 and 4 respectively.

Experimental results and analysis

In this section, we conducted performance evaluations on the data set presented in the ‘Dataset’ section. To ensure a fair evaluation of the models, we adopted the soft voting method for the final classification assessment. Our proposed models were compared with five traditional feature selection methods: mutual information (MI), F test (f_classif), ReliefF, SURF, and SURF*. First, we compared the impact of Spearman’s dimensionality reduction. Table 5 displays the number of features before and after dimensionality reduction for different datasets. We focused on evaluating Spearman’s optimization of dimensionality reduction using the jaffe dataset as an example. The evaluation metrics used were confusion matrix, accuracy, precision, recall, F1-score, and running time. Figures 5 and 6 depict the confusion matrix before and after dimensionality reduction. The experimental results are presented in Tables 6 and 7, which respectively display the evaluation metrics for different feature selection methods on the jaffe dataset before and after dimensionality reduction. Table 8 shows the improvement in various metrics after dimensionality reduction, comparing the results with and without dimensionality reduction for different feature selection methods. Next, we compared the accuracy, recall rate, and F1-score of the five traditional feature selection methods after Spearman’s dimensionality reduction and EFS-GINI on the aforementioned eight datasets. The results of our experiments on these models are presented in line charts in Figs. 7, 8, 9 and 10. Lastly, we assessed the improvement of EFS-GINI on various metrics compared to the five traditional feature selection methods, as shown in Table 9.

First, we assess the effectiveness of Spearman dimensionality reduction. As shown in Tables 6, 7 and 8, Spearman dimensionality reduction enhances the average accuracy, accuracy rate, recall rate, and F1-score of traditional feature selection methods (F test, mutual information, ReliefF, SURF, SURF*, and ESF-GINI) to varying degrees. Moreover, it significantly reduces the running time and improves feature selection efficiency by eliminating linearly dependent redundant features. The impact of Spearman dimensionality reduction on the Jaffe dataset can be observed in the confusion matrices presented in Figs. 5 and 6, where it notably reduces prediction errors.

Next, we evaluate the effectiveness of EFS-GINI in feature selection. The accuracy, accuracy rate, recall rate, and F1-score line charts demonstrate that EFS-GINI outperforms the other five methods on most datasets. However, it exhibits limited advantages or average performance when dealing with datasets with a small number of samples and classifications, such as lung, Prostate_GE, and TOX_171 datasets. Conversely, EFS-GINI excels in high-dimensional feature selection with a large sample size. An example from the Jaffe dataset shows that EFS-GINI improves accuracy, accuracy rate, recall, and F1-score compared to the five traditional feature selection methods. Furthermore, EFS-GINI exhibits significantly faster running time compared to ReliefF, SURF, and SURF* methods. Thus, the advantages of EFS-GINI in high-dimensional datasets are evident.