A feature selection method utilizing path accumulation cost, redundancy minimization, and interaction maximization for the diagnosis of coronary heart disease

Related work

Recent advancements in machine learning have made it a powerful tool for the early diagnosis and prediction of CHD, particularly in handling high-dimensional medical datasets. Machine learning models, such as decision trees, random forests, and support vector machines, have been widely applied in feature selection for CHD prediction, as they are capable of capturing complex, non-linear relationships within the data.

Feature selection methodologies can be broadly categorized into filter, wrapper, and embedded approaches (Zeng et al., 2015). In early CHD prediction, wrapper and embedded methods have gained prominence. Lin et al. (2022) employed a combination of gradient boosting tree feature selection and recursive feature elimination to predict acute coronary syndrome risk, successfully extracting 25 key variables from 430 complex medical features, culminating in an impressive 98.8% accuracy in ACS risk prediction. Similarly, Javeed et al. (2019) integrated a random search algorithm (RSA) with an optimized random forest (RF) model, achieving a commendable 93.33% accuracy on the Cleveland datase. Mohamed et al. (2020) developed a novel Parasite-Predator Algorithm (PPA) by amalgamating the strengths of Cat Swarm Optimization (CSO), Cuckoo Search (CS), and Crow Search Algorithm (CSA). After PPA, fitness is used to select the feature subset that maximizes the accuracy and minimizes the number of selected features. This approach achieved an 86.17% accuracy on the Statlog dataset, ultimately selecting only four key features (Mohamed et al., 2020).

(1) $$\mathrm{F}\mathrm{i}\mathrm{t}\mathrm{n}\mathrm{e}\mathrm{s}\mathrm{s}=\mathrm{m}\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}\left(\mathrm{A}\mathrm{c}\mathrm{c}+w_f\cdot \left(1-\frac{L_f}{L_t}\right)\right)$$where $\mathrm{A}\mathrm{c}\mathrm{c}$ is the classification accuracy, $L_f$ is the length of the selected feature subset, and $L_t$ is the total number of features.

Additionally, Bharti et al. (2021) conducted experiments on the UCI Heart dataset and found that combining feature selection with outlier detection yielded the best results. They applied LASSO for feature selection and trained models using machine learning and deep learning techniques. In the end, the deep learning model achieved an accuracy of 94.2%, while the KNN model achieved 84.8% (Bharti et al., 2021). Fajri, Wiharto & Suryani (2023) proposed a hybrid feature selection method combining Bee Swarm Optimization (BSO) and Q-learning (QBSO-FS), which improved heart disease detection accuracy, achieving 84.86% accuracy under the KNeighbors model, with faster convergence and better performance than traditional methods.

(2) $$F{C}_s=F_s-\theta C_s$$where $F{C}_s$ is the evaluation value of feature selection, $F_s$ is the accuracy of feature selection, $C_s$ is the cost of feature subset, and $\theta$ is the adjustment coefficient.

Despite their targeted nature, wrapper and embedded methods have obvious limitations. Particularly, wrapper methods require extensive model training and evaluation in the search for an optimal feature subset, a process that becomes especially laborious in scenarios involving large datasets or an abundance of features. Conversely, embedded methods heavily rely on specific models, which may limit their generalizability.

In contrast, filter methods, which rapidly identify features related to the target variable through statistical theory and are computationally simple and independent of any machine classifier model, have emerged as a practical and efficient alternative for managing complex medical datasets, such as those used in early screening prediction for CHD. Muhammad et al. (2020) introduced a machine learning-based CHD detection model, employing four feature selection techniques with ten classification algorithms, significantly improving classification accuracy. Reddy et al. (2023) enhanced the performance of various classifiers in cardiac risk prediction by combining the Cleveland and Statlog heart datasets and employing principal component analysis (PCA) and correlation-based feature selection (CFS). Peng, Long & Ding (2005) proposed the mRMR method based on information theory, aimed at using mutual information between features to discern relevant and redundant features. Yu & Liu (2003) introduced a FCBF method, focusing on selecting main features and eliminating highly redundant features through the computation of SU and Markov blankets.

However, these filter methods primarily evaluate the association of individual features with classification, disregarding the synergistic impact of combined features on the prediction. This oversight could potentially lead to the erroneous exclusion of important features or the retention of irrelevant ones, particularly in complex applications like CHD risk prediction. To address this challenge, Fleuret (2004), Bennasar, Hicks & Setchi (2015), Zeng et al. (2015), and Wang, Jiang & Jiang (2021) introduced various feature selection criteria (CMIM, JMIM, NJMIM, IWFS, MRMI), focusing on enhancing feature relevance, interaction, and redundancy removal, significantly improving the quality of feature selection and prediction accuracy. These advancements have collectively enhanced the quality of feature selection and prediction accuracy to a considerable extent. Notwithstanding the success of the above methods, they do not fully consider the relationships among all existing features and candidate features, potentially leading to the exclusion of relevant features or the inclusion of too many redundant features.

Contribution

Existing early screening predictive models for coronary heart disease (CHD) often struggle with managing the intricate relationships between existing and candidate features. This issue may lead to the omission of important features or the inclusion of unnecessary redundancies. To address this issue, this article introduces an early screening predictive model based on a novel CHD feature selection algorithm. Our model emphasizes the interactions among multiple diagnostic markers and their linkage to CHD. Further, it selects a feature subset where the strong associations between features yield more valuable information combinations, overcoming the above limitations and laying the groundwork for developing more sophisticated and effective CHD screening models. The main contributions are as follows:

• 1)

  We propose a feature selection approach based on soft path accumulative cost. This approach identifies the feature with the highest interaction by calculating the interaction of each current feature with all previous features. The final outcome is a subset of features with the strongest interrelationships, which best represent CHD.

• 2)

  To comprehensively evaluate the interactions among multiple features, we propose a novel interaction metric, Rel, and an accumulative interaction evaluation function SPA that measures the relationships between all existing features and the current candidate feature.

• 3)

  We introduce a novel stop criterion evaluation function to balance model evaluation performance and feature subset size. This function determines when to halt feature selection by measuring the increase in information gain and setting hyperparameters to balance the number of features with the final model evaluation performance.

The article is structured as follows: “Prior Knowledge of Feature Selection” introduces the background theoretical knowledge relevant to our discussion; “Methods” outlines the development process of our proposed early screening predictive model; “Results” describes the experimental setup and analyzes the findings. Finally, in “Discussion and Conclusion” we examines the implications of our study for improving CHD early screening and proposes directions for future advancements.

An information-theoretic perspective on feature selection

The fundamental goal of feature selection is to extract a key subset from the original feature set such that a classifier trained on this subset achieves classification performance comparable to, or even better than, that obtained using the full feature set. This goal can be formalized by the following conceptual function: under the constraint of classification performance, one seeks the smallest feature subset. That is,

(8) $$\underset{S\subseteq \mathcal{F}}{min}|S|\mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}\mathrm{f}(\mathrm{S})\ge \mathrm{f}(\mathcal{F})-\epsilon ,$$where $f(S)$ denotes the accuracy of a classifier trained on the subset S and $\epsilon$ represents the permissible degradation in performance. This formulation not only encapsulates the requirement to maintain performance close to that of the full feature set but also underscores the objectives of dimensionality reduction and redundancy minimization.

Filter-based feature selection methods grounded in information theory assess two key aspects: the redundancy among features and the relevance of each feature to the target labels. Traditionally, mutual information (MI) is used for both purposes, with $I(F_i,F_j)$ measuring the redundancy between features $F_i$ and $F_j$, and $I(F_i,C)$ quantifying the relevance between a feature $F_i$ and the target C. However, MI’s inherent limitations in scale and normalization have led to the adoption of symmetric uncertainty (SU) as a normalization tool, thereby enabling more consistent comparisons. Despite these improvements, the use of bivariate MI remains insufficient when trying to capture the additional joint gain achieved by combining a candidate feature with an already selected feature to explain the target. This deficiency motivates the introduction of three-way interaction mutual information, which is designed to account for the cooperative contribution of feature pairs in relation to the target.

The three-way interaction mutual information, denoted by $I(F_i,F_j,C)$, quantifies the additional information provided by the candidate feature $F_j$ in the presence of a selected feature $F_i$ when explaining the target C. Nevertheless, directly using this three-variable measure presents two issues. On one hand, it does not distinguish the redundancy that may exist between $F_i$ and $F_j$. On the other hand, the candidate feature $F_j$ might already exhibit high direct relevance to the target C, leading to potential double counting of its contribution.

To address these issues, we introduce two modulation terms based on symmetric uncertainty. The first modulation term is

(9) $$\frac{I(F_i,F_j,C)}{SU(F_i,F_j)},$$which adjusts for the redundancy between $F_i$ and $F_j$. A high value of $SU(F_i,F_j)$ indicates substantial redundancy between the two features, thereby reducing the value of this term and reflecting that a large portion of the joint information is redundant.

The second modulation term is

(10) $$\frac{I(F_i,F_j,C)}{SU(F_j,C)},$$which accounts for the direct relevance of the candidate feature $F_j$ to the target C. When $F_j$ is highly correlated with C, the high value of $SU(F_j,C)$ implies that much of the joint information is already captured by $F_j$’s individual relevance, and this contribution should not be overemphasized in assessing its unique gain.

Compared to conventional methods that rely solely on bivariate MI or SU to evaluate feature relevance and redundancy, this approach—employing three-way interaction mutual information modulated by SU offers a more comprehensive framework. It enables a refined assessment of a candidate feature’s joint gain, facilitating the selection of a compact feature subset that both preserves or enhances classification performance and reduces redundancy.

Quantitative evaluation of feature interactions and SPA evaluation function

From Eq. (4), it is evident that a newly added feature $F_j$, when combined with a previously selected best feature, should provide more information than each feature individually. However, as described in the section An Information-Theoretic Perspective on Feature Selection, highly correlated overlapping features might introduce excessive redundancy, which can be mitigated using symmetrical uncertainty (SU) to balance redundancy and relevance. Specifically, for a candidate feature $F_j$ and an already selected feature $F_i$, if $\mathrm{S}\mathrm{U}(F_j,C)<\mathrm{S}\mathrm{U}(F_i,F_j)$, this indicates that the relationship between $F_j$ and $F_i$ is stronger than that between $F_j$ and the label C. In such cases, $F_j$ and $F_i$ are considered to be highly redundant, indicating an overlap of information between $F_j$ and $F_i$.

Building on the characteristics of SU, we further propose an interaction metric, $\mathrm{R}\mathrm{e}\mathrm{l}(F_i,F_j,C)$. $\mathrm{R}\mathrm{e}\mathrm{l}(F_i,F_j,C)$ is defined as:

(11) $$\mathrm{R}\mathrm{e}\mathrm{l}(F_i,F_j,C)=\frac{I(F_i,F_j,C)}{SU(F_i,F_j)}-\frac{I(F_i,F_j,C)}{SU(F_j,C)}.$$

This metric evaluates the interactions between features by considering the three-way mutual information $I(F_i,F_j,C)$ and their symmetrical uncertainty SU, aiding in identifying feature combinations that may contribute limited predictive efficacy due to high redundancy.

Through $\mathrm{R}\mathrm{e}\mathrm{l}(F_i,F_j,C)$, we can identify features $F_j$ that are highly related to $F_i$ yet exhibit low redundancy. However, focusing solely on $\mathrm{R}\mathrm{e}\mathrm{l}(F_i,F_j,C)$ may lead to an overemphasis on the relationships among features without adequately considering the direct relevance of the candidate feature $F_j$ to the label C. Neglecting the direct association between candidate features and the label may result in a selected feature set that fails to establish effective predictive relationships, thereby impacting the model’s predictive performance.

To address this issue, the SPA evaluation function is designed to consider both the interactions among features and their direct information contributions to the label. Specifically, the SPA scoring criterion incorporates the interaction measure $\mathrm{R}\mathrm{e}\mathrm{l}(F_i,F_j,C)$ and explicitly includes the information contribution measure between the candidate feature $F_j$ and the label C. The SPA function evaluates the composite effect of a candidate feature on the entire best feature subset by cumulatively calculating the weighted interactions between the candidate feature and all the features already selected in the best subset. The SPA scoring criterion is defined as:

(12) $$\mathrm{S}\mathrm{P}\mathrm{A}=\sum _{F_i\in \mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t}}\sqrt{\mathrm{S}\mathrm{U}(F_j,C)}\cdot \mathrm{R}\mathrm{e}\mathrm{l}(F_i,F_j,C)$$where best is the subset of already selected best features, $F_j$ is the candidate feature, and $F_i$ is a previously selected best feature.

Based on the SPA evaluation function, we defined the SPFS selection strategy as follows:

(13) $$\mathrm{S}\mathrm{P}\mathrm{F}\mathrm{S}=\arg \underset{F_j\in \mathrm{c}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{e}}{max}\left(\sum _{F_i\in \mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t}}\sqrt{\mathrm{S}\mathrm{U}(F_j,C)}\cdot \mathrm{R}\mathrm{e}\mathrm{l}(F_i,F_j,C)\right)$$where candidate is the subset of candidate features. Through the SPFS scoring criterion, by considering the relevance of the current candidate feature $F_j$ to the label C (weighted by $\mathrm{S}\mathrm{U}(F_j,C)$), we accumulate the interactive effects between the candidate feature and all previously selected features to select the candidate feature with the highest interaction with any feature in the best subset. This means that for the selected best features, the chosen candidate feature must demonstrate an unparalleled interaction with at least one feature in the best subset.

Stop scoring function

Given the endless process of finding the highest interacting features, a reasonable stop scoring function is needed to identify an optimal feature subset that balances the number of features and prediction accuracy. We proposed that a feature should be included in the best subset if its addition significantly increased the information gain of the subset. The information gain ratio is extensively used to differentiate the contributions of features, because it reflects both the feature’s entropy and the information gain contribution to the target variable, allowing us to precisely evaluate the effectiveness of different feature subsets.

The $\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}$ is defined in Eq. (9), where $g_R(D,\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t})$ and $g_R(D,\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{\_}\mathrm{b}\mathrm{e}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{e})$ are the information gain ratios of the current best feature subset and the previous best feature subset for the entire dataset D, respectively.

(14) $$\begin{array}{rl}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=& g_R(D,\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t})-g_R(D,\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{\_}\mathrm{b}\mathrm{e}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{e})\\ =& \frac{g(D,\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t})}{H_{\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t}}(D)}-\frac{g(D,\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{\_}\mathrm{b}\mathrm{e}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{e})}{H_{\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{\_}\mathrm{b}\mathrm{e}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{e}}(D)}.\end{array}$$

The $\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}$ is calculated to determine whether the current best feature subset provides a more significant information gain boost compared to the previous best feature subset. The conditions are as follows:

(15) $$\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\Rightarrow \{\begin{array}{cc}\mathrm{T}\mathrm{h}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{w}\mathrm{l}\mathrm{y}\mathrm{s}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}\mathrm{b}\mathrm{r}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{s}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n},\hfill & \mathrm{i}\mathrm{f}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}<0,\hfill \\ \mathrm{T}\mathrm{h}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{w}\mathrm{l}\mathrm{y}\mathrm{s}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}\mathrm{b}\mathrm{r}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{s}\mathrm{b}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n},\hfill & \mathrm{i}\mathrm{f}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}>0.\hfill \end{array}$$

More precisely, Eq. (12) is explained in detail as follows:

• If $\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}<0$, it indicates that the newly selected feature does not aid in the prediction accuracy and only increases the feature complexity.

• If $\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}>0$, it suggests that the newly selected feature brings more beneficial information, aiding the model in making accurate predictions.

However, in some cases, the newly selected feature may not initially provide a significant boost in information gain, but as subsequent features are combined, it may produce better effects. To address this, we introduced a hyperparameter k_stale to delay the stop. When the number of times $\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}<0$ reaches the threshold specified by k_stale, the selection of new features will be stopped. This approach allows for more flexible handling of early stopping issues caused by local minima in information gain.

Procedures of soft path feature selection

As illustrated in Algorithm 1, given a dataset D containing $n$ samples, $m$ features, and a class feature C, the SPFS algorithm identifies the best feature subset best. The SPFS feature selection process involves four primary stages: initialization, k_stale tuning, pre-computation, and feature selection.

In the initialization stage, the candidate feature subset, the best feature subset, and relevant algorithm parameters are initialized. During the pre-computation stage, the SU values for each feature and the mutual interaction Rel table between every pair of features and the label are calculated. The features are then ranked in descending order of their SU values to form the candidate feature subset.

The third stage focuses on tuning the hyperparameter $k\mathrm{\_}stale$. Each $k\mathrm{\_}stale\mathrm{\_}value$ in $k\mathrm{\_}range$ is applied during the feature selection phase, and the performance of each is evaluated to select the best $k\mathrm{\_}stale$ for the final feature selection stage.

The fourth stage is a key step in the SPFS algorithm, and it’s divided into two steps:

1. Select individual features: First, when the best feature subset is empty, the top-ranked feature from the sorted candidate feature subset is selected as the first feature of the best feature subset. In each iteration, the SPA evaluation function accumulates the interaction score of each candidate feature $F_i$ with the selected subset. Then, by using the SPFS selection strategy, the highest-scoring $F_{\mathrm{b}\mathrm{e}\mathrm{s}\mathrm{t}}$ feature is chosen as the new feature in the best subset.

2. Select the final best feature subset: After each feature selection, the score is calculated; if the score is positive, iteration continues. When a negative score first appears, iteration continues, but it enters a delayed stopping step, counting the number of times the score is below zero and recording the best subset selected. When the number of times the score is below zero exceeds $k\mathrm{\_}stale$, the addition of new features stops, and the algorithm selects the highest-scoring feature subset from these $k\mathrm{\_}stale$ candidate best subsets as the final determined best subset.

Moreover, our algorithm also focuses on the stability of features. By observing whether the selected features remain consistent across multiple iterations and different data partitions, the stability of the feature selection process is assessed. A highly stable feature selection process indicates that the selected features have a genuine and reliable association with the target variable.

Ultimately, the SPFS algorithm identifies an optimal feature subset best that is not overly complex but can effectively predict the class feature C, derived after considering feature importance, interaction, and feature stability. Through this approach, the algorithm not only enhances the prediction accuracy of the model but also avoids unnecessary model complexity, thereby providing an efficient and reliable feature selection solution for practical applications.

Experiment setup

This research is dedicated to the development of an early screening and prediction model for coronary heart disease (CHD) that is particularly tailored for the young and middle-aged demographic, aiming to minimize the number of diagnostic markers needed. For this purpose, we analyzed four datasets associated with cardiac health and CHD. The first dataset, Heart, from the UCI Machine Learning Repository, features simplified binary classification of heart disease severity and excludes records with missing values. The Z-Alizadeh Sani dataset contains data from 303 patients at the Shaheed Rajaei Cardiovascular Center (Alizadehsani et al., 2013). Following this, Cardiovascular (Lin et al., 2022), from a tertiary hospital in Fujian, China, includes 2,702 patient records and focuses on predicting all-cause mortality, with all data anonymized. Lastly, CHD (Cao et al., 2022), approved by the ethics committee and anonymized, contains records from 715 patients at a tertiary hospital in Anhui, China. For both the Cardiovascular and Coronary Heart Disease datasets, they are open-source datasets from published articles that have already undergone ethical review in the original studies, and thus, this research does not require additional ethical approval.

Table 1 details the datasets used, including the number of features, cases, and categories. In the data preprocessing stage, since the obtained datasets had already undergone certain preprocessing steps by their original authors, we only implemented discretization to the datasets. Discretization aims to convert continuous numerical variables into discrete categorical values (such as 0 and 1), facilitating subsequent feature selection and classification prediction. Specifically, we utilized the Minimum Description Length Principle (MDLP), a widely recognized discretization method proposed by Fayyad & Irani (1993), to discretize all continuous numerical features. Additionally, for the imbalanced Cardiovascular dataset, the SMOTE technique was applied to address data imbalance issues during the classification stage, consistent with the methodology described in the original article. The implementation code for SPFS can be found at https://github.com/YUkiJiang559/SPFS.

To validate the efficacy of the proposed method, this study compared various widely used supervised feature selection algorithms, including FCBF (Yu & Liu, 2003), Consistency (Dash & Liu, 2003), mRMR (Peng, Long & Ding, 2005), CFS (Hall, 2000), Relief-F (Urbanowicz et al., 2018), CMIM (Fleuret, 2004), JMIM (Bennasar, Hicks & Setchi, 2015), IWFS (Zeng et al., 2015), and MRMI (Wang, Jiang & Jiang, 2021). These algorithms aim to identify the most relevant features, eliminate redundancies, and consider interactions among features to obtain a best feature subset. If the original articles of the datasets provided related feature selection results, these were also included for comparison. Among them, the CFS, FCBF, and mRMR algorithms obtain the best feature subset by finding the most relevant features and removing redundant features; the CMIM, JMIM, and IWFS algorithms focus more on the interaction between features to obtain the best feature subset, while MRMI uses the interaction between features while considering the relevance and redundancy of the features. Specifically, to ensure a fair comparison with the SPFS algorithm, CFS, Consistency, FCBF, and Relief-F were implemented using Weka’s built-in library with default parameters (Eibe Frank & Witten, 2016). Similarly, CMIM, JMIM, and IWFS were implemented through the ITMO-FS library with default parameter settings (Computer Technologies Laboratory, 2024). It should be noted that the feature selection methods chosen for this study represent a set of well-established, conventional techniques that have been widely adopted in high-quality research publications as standard benchmarks for comparing feature selection performance.

Empirical results

In line with the experimental design outlined earlier, we evaluated the performance of feature selection algorithms by examining the number of selected features and their classification accuracy. This evaluation involved classifying the best feature subsets with XgBoost, naive Bayes, Linear-SVM, and random forest algorithms, all configured with default parameter settings (Chen & Guestrin, 2016; Cervantes et al., 2020; Antoniadis, Lambert-Lacroix & Poggi, 2021; van de Schoot et al., 2021). To ensure the stability and accuracy of the prediction results, ten-fold cross-validation was applied across all models. And two metrics were used to quantitative evaluation of feature selection algorithms: the algorithm’s classification accuracy and its effectiveness in reducing feature dimensionality. Ideally, an algorithm that significantly surpasses others in classification accuracy is considered to have the best performance. However, when the difference in classification accuracy among algorithms is not significant, those that more effectively reduce feature dimensionality are deemed superior.

Among the involved algorithms, mRMR, Relief-F, CMIM, JMIM, and IWFS use a ranking procedure to select features, while others select the best feature subset based on their specific stopping criteria. Therefore, this study’s experimental design aims to compare feature selection algorithms in two directions: firstly, for those algorithms determining the best feature subset based on their stopping criteria (CFS, Consistency, and FCBF), we compare their performance in reducing feature dimensions and the classification accuracy of their selected feature subset; secondly, for rank-based algorithms (mRMR, Relief-F, CMIM, JMIM, and IWFS), we evaluate the incremental gain in classification prediction accuracy for each feature selected in rank order. Considering the MRMI algorithm and the SPFS algorithm include both feature ranking and specific stopping criteria, this study evaluates their performance in both directions.

Performance comparison of mRMR, Relief-F, CMIM, JMIM, IWFS, and MRMI

For algorithms that select features based on a ranking procedure, by specifying a required number of features N, the feature selection algorithm outputs the top N best features to form the final most important feature ranking subset. For the four datasets covered in the experiment, the required number of features N was set to 10. For these datasets, each algorithm ranked the features by importance and provided the corresponding most important feature ranking subset. Table 5 lists the different algorithms selected features in order of importance. We can observe that although the ranking orders vary, certain key features consistently appear at the top across the subsets of different algorithms. These commonly deemed important features form the core of the SPFS algorithm’s selection. This indicates that while each feature selection algorithm can capture the most important information among many features, it may also capture incorrect information or noise. Meanwhile, the SPFS algorithm exhibits significant efficiency in feature selection, adeptly identifying the most crucial features within complex information while disregarding irrelevant ones. This underscores the SPFS algorithm’s effectiveness in capturing essential data within datasets.

Table 5:

Top-10 ranked features by different algorithms.

Methods	Heart	Z-Alizadeh Sani	Cardiovascular	Coronary heart disease
mRMR	13, 12, 3, 11, 9, 2, 8, 10, 1, 4	25, 54, 1, 35, 53, 6, 28, 18, 24, 32	75, 49, 69, 29, 45, 72, 26, 81, 56, 74	9, 41, 22, 28, 16, 43, 29, 1, 23, 33
Relief-F	13, 2, 9, 12, 8, 3, 11, 10, 6, 1	25, 28, 7, 53, 26, 8, 1, 35, 4, 6	10, 56, 57, 40, 5, 4, 6, 8, 3, 2	22, 41, 1, 12, 28, 43, 16, 18, 38, 11
CMIM	13, 12, 3, 9, 11, 2, 10, 8, 7, 6	25, 54, 35, 53, 18, 6, 1, 55, 32, 38	75, 49, 45, 20, 29, 69, 26, 57, 44, 21	9, 41, 22, 28, 17, 16, 1, 33, 27, 18
JMIM	13, 12, 3, 9, 11, 2, 10, 8, 1, 7	25, 54, 28, 7, 53, 55, 35, 1, 29, 6	75, 49, 69, 45, 81, 71, 76, 70, 20, 29	9, 41, 22, 28, 43, 17, 23, 27, 16, 29
IWFS	13, 12, 3, 9, 10, 11, 2, 8, 1, 7	25, 54, 28, 53, 24, 35, 1, 29, 6, 32	74, 81, 32, 72, 75, 69, 80, 38, 22, 77	41, 22, 28, 9, 43, 29, 23, 33, 17, 16
MRMI	13, 9, 12, 8, 3, 10	25, 53, 39, 14, 15	74, 6, 26, 45, 29	41, 1, 30, 28, 29
SPFS	13, 9, 12	25, 53, 1	74, 6, 18, 71	41, 1, 22, 9, 28

DOI: 10.7717/peerj-cs.2834/table-5

The experimental results displayed in Fig. 2 indicate that the accuracy of almost all feature selection algorithms improves with an increase in the number of features. However, after a certain point, the performance improvement of the algorithms plateaus or even declines, highlighting the importance of selecting an optimal number of feature subsets to enhance model performance.

Comparing with algorithms that select features based on a ranking procedure, it can be seen that the results obtained using the SPFS method are very close or similar to those obtained with mRMR, Relief-F, CMIM, JMIM, IWFS, and MRMI methods, and sometimes even better.

In the Heart and Cardiovascular datasets, SPFS was almost the best-performing algorithm. Besides SPFS, other algorithms exhibited significant performance drops or sharp declines after initial improvements. The sudden decline and subsequent improvement in the performance of the SPFS algorithm on the Heart dataset may be because the second feature alone did not contribute much information. However, it has a strong mutual correlation with the third feature, leading to a significant increase in accuracy performance once the third feature was included. This can be attributed to the effectiveness of the $k\mathrm{\_}stale$ parameter, which prevents premature stopping, allowing the algorithm to continue selecting features that provide additional useful information when combined with previous ones. As can be seen by the SPFS without $k\mathrm{\_}stale$ in the line graph, Heart will stop at the second feature without $k\mathrm{\_}stale$. In the Z-Alizadeh Sani dataset, SPFS’s steady improvement was second only to the MRMR algorithm, while other algorithms required more features to achieve comparable or higher accuracy levels, contrary to our objective of reducing feature dimensions. In the Coronary Heart Disease dataset, the SPFS algorithm’s performance steadily increased with the addition of features. Although the final result was not the best, it was similar to the performances of other superior algorithms, which did not show much improvement or even deteriorated with additional features. In the Cardiovascular dataset, the $k\mathrm{\_}stale$ parameter effectively stopped the feature selection before accuracy declined sharply, ensuring that the selected feature subset remained highly effective.

In summary, the SPFS algorithm can select the most important and accurate features with a lower required number of features, achieving rapid improvements in accuracy while balancing high accuracy and low feature dimensionality. Comparisons with the other six feature selection methods show that SPFS can achieve better or similar performance with fewer features, proving its superiority.