Feature selection based on neighborhood rough sets and Gini index

Neighborhood rough set

In rough sets, information tables are often represented by $<U,A>$, where $U=\{x_1,x_2,...,x_n\}$ is a non-empty finite set of samples and $A=\{a_1,a_2,...,a_m\}$ is a non-empty finite set of attributes used to describe those samples.

Let $<U,A>$ be a table of information, $B\subseteq A$ and $d_B$ is a binary functional relation defined on $U$ with an attribute subset B, that is, $d_B:U\times U\to R^{+}$. Then, $d_B$ is said to be a distance metric on $U$ when it satisfies the following relation:

(1) $d_B(x_1,x_2)\ge 0$, $d_B=0$ if and only if $x_1=x_2$, $\mathrm{\forall }{x}_1,x_2\in U$;

(2) $d_B(x_1,x_2)=d_B(x_2,x_1)$, $\mathrm{\forall }{x}_1,x_2\in U$;

(3) $d_B(x_1,x_3)\le d_B(x_1,x_2)+d_B(x_2,x_3)$, $\mathrm{\forall }{x}_1,x_2,x_3\in U$.

The Euclidean distance is a commonly used distance measure, and all the subsequent distance references in this article are in terms of the Euclidean distance. For any two samples, the calculation of the Euclidean distance is as follows:

(1) $$d_B(x_i,x_j)=\sum _{a\in B}\sqrt{{(x_i^a-x_j^a)}^2}$$

In an information table $<U,A>$, for any sample $x\in U$, attribute subset $B\subseteq A$, the neighborhood similarity relation $R_B^{\sigma }$ is defined as follows:

(2) $$R_B^{\sigma }=\{(x,y)\in U\times U|d_B(x,y)\le \sigma \}$$where $\sigma \ge 0$ is a user-defined constant. For any $x\in U$, its neighborhood similarity class $[x{]}_B^{\sigma }$ is defined as follows:

(3) $$[x{]}_B^{\sigma }=\{y\in U:(x,y)\in R_B^{\sigma }\}$$

In an information system, neighborhood similarity classes are also referred to as neighborhood information granules, abbreviated as neighborhood granules. Here, $\sigma$ is called the radius of the neighborhood granules. In the information table $<U,A>$, the neighborhood granule family $\{[x_i{]}_B^{\sigma }|i=1,2,...,n\}$ forms a covering of $U$. The neighborhoods of all objects in the domain constitute the granulation of the domain, and the neighborhood granule family constitutes the fundamental concept system in the domain space. Through these fundamental concepts, we can approximate any concept in the space.

For any sample set $X\subset U$, its lower approximation ${\underset{\_}{R}}_B^{\sigma }$ and upper approximation ${\overline{R}}_B^{\sigma }$ are defined as follows:

(4) $${\underset{\_}{R}}_B^{\sigma }=\{x\in U:[x{]}_B^{\sigma }\subseteq X\}$$

(5) $${\overline{R}}_B^{\sigma }=\{x\in U:[x{]}_B^{\sigma }\cap X\ne \mathrm{Ø}\}$$

Let $D$ be a classification decision attribute defined on $U$, and $A\cap D=\mathrm{Ø}$. In this case, the triple $<U,A,D>$ is referred to as a decision table. In the decision table $<U,A,D>$, attribute D divides U into $r$ decision classes, denoted as $U/D=\{E_1,E_2,...,E_r\}$. Here, $E_i(i=1,2,...,r)$ is called a general equivalence class, meaning that all the samples in $E_i$ have the same class labels. In a decision table $<U,A,D>$, where $B\subseteq A$, and $R_B^{\sigma }$ is a neighborhood similarity relation defined on the attribute set $B$ in $U$ with a neighborhood radius of $\sigma$, the upper approximation ${\overline{R}}_B^{\sigma }(D)$ and lower approximation ${\underset{\_}{R}}_B^{\sigma }(D)$ of the decision attribute $D$ with respect to the attribute set $B$ at a neighborhood granule size of $\sigma$ are defined as follows:

(6) $${\overline{R}}_B^{\sigma }(D)={\cup }_{k=1}^r{\overline{R}}_B^{\sigma }(E_k)$$

(7) $${\underset{\_}{R}}_B^{\sigma }(D)={\cup }_{k=1}^r{\underset{\_}{R}}_B^{\sigma }(E_k)$$

The positive domain of the decision table is written as:

(8) $$PO{S}_B^{\sigma }(D)={\cup }_{E_k\in U/D}{\underset{\_}{R}}_B^{\sigma }(E_k)$$

The boundary domain of the decision table is written as:

(9) $$R{n}_B^{\sigma }(D)=U_B(D)-PO{S}_B^{\sigma }(D)$$

The dependency function ${\gamma }_B^{\sigma }(D)$ of D associated with B is formulated as:

(10) $${\gamma }_B^{\sigma }(D)={\displaystyle \frac{|PO{S}_B^{\sigma }(D)|}{|U|}}$$where $|.|$ indicates the cardinality of a set.

Gini index

GI is a metric used to measure the impurity of a dataset and is commonly employed in feature selection for decision tree algorithms. The values of GI range from 0 to 1. When GI = 0, the dataset’s impurity is minimal, meaning all elements in the dataset are the same. Conversely, when GI = 1, the dataset’s impurity is maximal, indicating that all elements in the dataset are different. For a dataset $D$ with $r$ categories, where each category’s proportion of samples is denoted as $p_i$, the formula for calculating GI is as follows:

(11) $$GI(D)=1-\sum _{i=1}^r{p}_i^2$$

GI evaluates the impurity of a dataset based on the distribution of class probabilities to determine the importance of the corresponding features. A smaller GI indicates higher dataset purity and better discriminative power of the feature. However, in the same dataset, different feature subsets result in the same class probability distribution, making it unsuitable for directly evaluating the classification performance of different feature subsets. Therefore, a new influencing factor must be introduced to make the class probability distributions vary across feature subsets. For instance, in the Classification and Regression Trees, a tree-like structure is introduced to partition the dataset. This partitioning leads to substantial differences between data subsets created by different feature divisions, resulting in distinct class probability distributions. This makes the Gini index enable the measurement of feature importance. In NRS, when the neighborhood radius is consistent, different attribute sets lead to distinct neighborhoods for samples. Conversely, when the attribute set is constant, various neighborhood radii correspond to different neighborhoods. Different neighborhoods could lead to varying class probability distributions. GI values would also differ. This makes GI applicable for feature selection in NRS.

Next, two different feature importance evaluation metrics integrating NRS and GI will be proposed.

The proposed method

Neighborhood Gini index

Samples with certain similarities should be grouped into the same category, and samples within the neighborhood range of a sample are considered similar from a distance perspective. Their features determine the similarity of samples. However, for reasons such as data collection, some features may be redundant or irrelevant to class labels. Therefore, the class labels of samples within a neighborhood range may not be consistent under a subset of features. It is necessary to select features that can effectively represent the characteristics of all categories so that the class labels within the neighborhood range of samples are as consistent as possible.

The more consistent the class labels of samples within the neighborhood range, or the higher the purity of classes within the neighborhood range, the better the corresponding subset of features can represent that class. At this point, the subset of features can represent the local characteristics of that class well. If a subset of features can represent all the local characteristics of all classes well, i.e., the class purity within the neighborhood range of all samples in the dataset is high, then the subset of features can distinguish all classes well. In this case, the importance of features is also higher.

Based on this idea, we use GI to represent the impurity of the dataset and propose the Neighborhood Gini index (NGI). NGI evaluates the importance of a subset of features by assessing the purity of all samples’ neighborhood ranges under that feature subset. The definition of NGI is given below:

Definition 1: Given a decision table $<U,A,D>$, for any $x_i\in U$, $B\subseteq A$, the impurity of $\sigma$ neighborhood $R_B^{\sigma }(x_i)$ is defined as:

(12) $$G{I}_B^{\sigma }(x_i)=1-\sum _{j=1}^r{p}_j^2$$where $r$ represents the number of categories, and $p_j$ signifies the proportion of the jth category within the $\sigma$ neighborhood of $x_i$.

Throughout the decision table, the impurity of the decision table is the mean value of the impurity within the neighborhood of each sample:

(13) $$NG{I}_B^{\sigma }(D)={\displaystyle \frac{1}{n}\sum _{i=1}^{n}G{I}_B^{\sigma }(x_i)}$$

From Definition 1, it can be observed that NGI is influenced by two parameters: the feature subset $B$ and the neighborhood radius $\sigma$. As GI focuses on the distribution of classes, changes in the number of samples within the neighborhood range can lead to class distribution changes, thereby affecting GI’s magnitude. However, the behavior of GI to changes in the feature subset and neighborhood radius is not strictly monotonic. The following will analyze the variations of NGI to changes in the feature subset and neighborhood radius separately.

Impact of feature on NGI

For any subset of features $B_1\subset B_2\subseteq A$, adding one or more features to $B_1$ to obtain $B_2$ does not necessarily guarantee that $NG{I}_{B_2}(D)$ will always be smaller than $NG{I}_{B_1}(D)$, and the process of its change is not completely monotonous, as shown in Fig. 1.

Figure 1 shows the relationship between different sizes of feature subsets and NGI under the same neighborhood. The x-axis is the number of features, and the y-axis is the NGI of the corresponding feature. The smaller feature subset is a proper subset of the larger feature subset. In Fig. 1A, the feature subset consists of 18 continuous features, namely [17, 59, 1, 18, 51, 30, 21, 10, 52, 20, 50, 39, 53, 55, 29, 54, 48, 25], and the data is the “Sonar” dataset from the UCI Machine Learning Repository (Kelly, Longjohn & Nottingham, 1998). The neighborhood radius in Fig. 1A is set to 0.15. In Fig. 1B, the feature subset comprises 18 discrete features, namely [4, 7, 15, 10, 9, 13, 14, 6, 17, 16, 0, 12, 3, 2, 5, 8, 11, 1], and the data is the “anneal” dataset from the UCI Machine Learning Repository. The neighborhood radius in Fig. 1B is set to 0.4.

When the features are continuous, the variation in the samples within the neighborhood range is small, leading to minor changes in the class distribution. As a result, the variation curve is relatively smooth, as shown in Fig. 1A. However, when the features are discrete, introducing new features might drastically reduce the number of samples within the neighborhood range, causing larger changes in the class distribution. This results in a fluctuating variation curve, as depicted in Fig. 1B.

From Fig. 1, it is evident that the variation of NGI is not strictly monotonic at a local level, yet it generally exhibits a descending trend as a whole. This phenomenon can be attributed to the overarching effect that, with an increase in the number of features, the data provides a more precise portrayal of the samples, making their inherent characteristics more prominent. When the features a sample emphasizes are more aligned with its class attributes, the purity of the sample’s neighborhood increases, leading to a smaller NGI. Conversely, when the emphasized features deviate from the class attributes, the neighborhood’s purity decreases, resulting in a larger NGI. As the number of features expands, characteristics relevant to the class gradually come into sharper focus, consequently contributing to the observed overall decreasing trend.

It is worth noting that not all continuous feature subsets follow smooth and monotonic variation curves, and not all discrete features yield fluctuating curves. Continuous features might also exhibit fluctuations, while discrete features can exhibit smooth and monotonic behaviors. However, regardless of whether the features are continuous or discrete, the overall tendency is characterized by a decrease.

The subsequent explanation illustrates the variation of NGI through changes in the class distribution within the feature space neighborhood of sample $x_i$:

We utilize the change in GI within the neighborhood feature subspace of sample $x_i$ to symbolize the overall changes in NGI across the entire dataset. The distribution of samples within this localized neighborhood feature subspace is depicted in Fig. 2. Among them, the hollow circle class accounts for ${\displaystyle \frac{1}{3}}$, and the solid circle class accounts for ${\displaystyle \frac{2}{3}}$. At this time, $NG{I}_B^{\sigma }(x_i)=1-{\displaystyle {\frac{1}{3}}^2}-{\displaystyle {\frac{2}{3}}^2}={\displaystyle \frac{4}{9}}$. Let $a_i\in A-B$, $NG{I}_{B\cup \{a_i\}}^{\sigma }$ change relative to $NG{I}_B^{\sigma }$ as follows:

1. NGI increases when the number of samples in the neighborhood feature subspace with a large proportion of categories decreases proportionally more than the number of samples with a small proportion of categories. As shown in Fig. 3A, the number of samples in the hollow circle category decreases by 1, at which point the percentage is ${\displaystyle \frac{2}{4}}$, and the number of samples in the solid circle category decreases by 4, at which point the percentage is ${\displaystyle \frac{2}{4}}$, $NG{I}_B^{\sigma }={\displaystyle \frac{4}{9}<NG{I}_{B\cup \{a_1\}}^{\sigma }=1-{\displaystyle {\frac{2}{4}}^2}-{\displaystyle \frac{2}{4}={\displaystyle \frac{1}{2}}}}$;

2. NGI remains unchanged when there is no change in the samples in the neighborhood feature subspace or when the samples in the neighborhood, according to the proportion of categories in equal, are reduced. As shown in Fig. 3B, at this time the proportion of the hollow circle category is still ${\displaystyle \frac{1}{3}}$, and the proportion of the solid circle category is ${\displaystyle \frac{2}{3}}$, $NG{I}_{B\cup \{a_2\}}^{\sigma }=NG{I}_B^{\sigma }={\displaystyle \frac{4}{9}}$;

3. NGI decreases when the number of samples in the neighborhood feature subspace with a large proportion of categories decreases proportionally less than the number of samples with a small proportion of categories. As shown in Fig. 3C, the hollow circle category samples are reduced by 2, at this time the proportion of ${\displaystyle \frac{1}{5}}$, the solid circle category samples are reduced by 2, at this time the proportion of ${\displaystyle \frac{4}{5}}$, $NG{I}_B^{\sigma }={\displaystyle \frac{4}{9}>NG{I}_{B\cup \{a_3\}}^{\sigma }=1-{\displaystyle {\frac{1}{5}}^2}-{\displaystyle {\frac{4}{5}}^2}={\displaystyle \frac{8}{25}}}$.

Impact of neighborhood radius on NGI

In addition to the influence of feature subsets on NGI, the size of the neighborhood radius also affects the changes in the distribution of classes within the neighborhood feature subspace, consequently impacting the magnitude of NGI. So, we have delved into the impact of varying neighborhood radius sizes on NGI. We set the neighborhood radius to range from 0 to 1 with a step size of 0.025, and the relationship between the neighborhood radius and NGI is depicted in Fig. 4.

Figure 4 illustrates the relationship between different sizes of the neighborhood radius and NGI for the same feature subset. The x-axis is the size of the neighborhood radius, and the y-axis is the NGI of the corresponding neighborhood radius. In Fig. 4A, the feature subset consists of 10 continuous features, namely [46, 8, 3, 2, 44, 53, 59, 24, 25, 42], sourced from the “Sonar” dataset in the UCI Machine Learning Repository. In Fig. 4B, the feature subset comprises five discrete features, namely [7, 14, 10, 1, 17], sourced from the “anneal” dataset in the UCI Machine Learning Repository.

As the value of $\sigma$ gradually increases, the number of samples within the neighborhood range also increases, leading to a rise in impurity. When $\sigma$ is relatively small, the change in the number of samples within the neighborhood is small, and the newly added samples are mostly from the same category. Consequently, the change curve remains relatively stable. When $\sigma$ exceeds a certain threshold (as depicted in Fig. 4A, e.g., 0.195), the category labels of the newly added samples start to deviate from those of the original samples. This leads to a change in NGI, eventually converging to the GI of the entire dataset. While an overall trend increases as the neighborhood radius gradually enlarges, this change is not necessarily monotonic. The reasons behind the variation of NGI with $\sigma$ are analogous to the reasons for its variation with the size of the feature subset. These reasons will not be reiterated here.

Neighborhood Class Gini index

In the context of neighborhood rough sets based on decision tables, the upper approximation of a category refers to the set of samples within the neighborhood range of that category. This sample set includes all samples from the current category and some from others. It is obvious that the fewer categories in the upper approximation and the fewer samples from other categories, the higher the purity of the upper approximation of the category. A higher purity indicates that the corresponding feature provides a more accurate description of that category, making it easier to distinguish it from others. If the upper approximations of all categories have higher purity, all types within the dataset can be better distinguished, and the corresponding features are more important. Based on this principle, this article proposes the Neighborhood Class Gini index (NCGI). It evaluates features’ importance by assessing the upper approximation’s impurity under different feature subsets. The definition of NCGI is provided below:

Definition 2: Given a decision table $<U,A,D>$, let $B\subseteq A$, $E_k\in U/D(k=1,2,...,r)$, and ${\overline{R}}_B^{\sigma }(E_k)$ is the upper approximations of $E_K$, so the impurity of ${\overline{R}}_B^{\sigma }(E_k)$ is defined as:

(14) $$G{I}_B^{\sigma }({\overline{R}}_B^{\sigma }(E_k))=1-\sum _{i=1}^r{p_i}^2$$

In the entire decision table, the impurity of the decision table is the average of the impurities of all category upper approximations:

(15) $$NCG{I}_B^{\sigma }(D)={\displaystyle \frac{1}{r}\sum _{k=1}^{r}G{I}_B^{\sigma }({\overline{R}}_B^{\sigma }(E_k))}$$

Similar to NGI, the magnitude of NCGI is also influenced by the neighborhood radius $\sigma$ and the feature subset $B$. The following comparison illustrates the changes in the two evaluation metrics concerning the number of features and the neighborhood radius. The data in Fig. 5 corresponds to the data in Fig. 1, while the data in Fig. 6 corresponds to that in Fig. 4.

In the case of continuous features, the trend of NCGI with changing $\sigma$ is closely similar to that of NGI, displaying relatively smooth changes. NCGI exhibits a lower overall sensitivity to variations in the number of features and the neighborhood radius yet displays higher sensitivity within certain intervals, such as when $\sigma$ ranges from 0.2 to 0.375 in Fig. 5A. This phenomenon stems from NGI being rooted in the sample neighborhood, with the class distribution altering as the neighborhood radius expands. Conversely, NCGI assesses feature importance from a class neighborhood perspective. As the neighborhood radius expands, the number of samples within the neighborhood increases. However, when $\sigma$ is small, the newly added samples within the neighborhood share the same category as the current sample. Consequently, the category distribution in the upper approximation remains unchanged. When $\sigma$ is big enough, the upper approximation of the class encompasses all samples within the dataset, resulting in NCGI equating to the overall GI and ceasing to change with variations in the neighborhood radius.

In the case of discrete data, as the neighborhood radius varies, a sudden influx of samples within the neighborhood range can significantly alter the class distribution, causing larger fluctuations in the change curve, particularly noticeable in Fig. 5B. However, overall, NCGI experiences smaller changes in amplitude compared to NGI.

The number of samples within the neighborhood range gradually decreases with increased features. In the context of continuous features, the reduction in sample count is relatively smooth, as shown in Fig. 6A. Consequently, the class distribution alteration of neighborhoods is similarly gradual. With the increase of purity within the neighborhood, GI decreases until it converges to 0.

For categorical features, the introduction of new features can exert a substantial influence on the class distribution within the neighborhood, leading to larger fluctuations, as shown in Fig. 6B. This is particularly evident upon the inclusion of the 11th feature, where both NGI and NCGI exhibit a sharp decline. This decline implies that adding this feature enhances the purity within the neighborhood, facilitating the differentiation of various categories. Beyond the 11th feature, sample neighborhoods and class neighborhoods’ results diverge. These features can decrease the impurity within the sample neighborhood but paradoxically lead to an increase in the impurity within the class neighborhood.

Feature selection

Definition 3: Given a decision table $<U,A,D>$, $B\subseteq A$, $a_i\in A-B$, the importance of $a_i$ with respect to $B$ is calculated as follow:

(16) $$SIG(a_i,B,D)=G{I}_B^{\sigma }(D)-G{I}_{B\cup \{a_i\}}^{\sigma }(D)$$where $G{I}_B^{\sigma }$ stands for NGI and NCGI proposed in this article.

In Definition 3, we defined the importance of feature $a_i$ relative to a given feature subset B. In the case where feature subset B is known, it is adding a feature $a_i$ to B and observing its GI (which refers to either NGI or NCGI). If GI decreases, it indicates that $a_i$ is a crucial feature relative to B. Conversely, if the GI remains unchanged or increases, it suggests that $a_i$ is a redundant feature relative to B or even an irrelevant feature with respect to the decision table $<U,A,D>$.

To achieve better classification performance, we aim to select each feature $a_i$ in such a way that it is the most crucial feature relative to B. Therefore, we have designed heuristic algorithm based on Neighborhood Gini index (HANGI) and heuristic algorithm based on Neighborhood Gini index (HANCGI) feature selection algorithms using a forward greedy approach to select the optimal feature subset. The two algorithms differ only in calculating $SIG(a_i,B,D)$, and their processes are illustrated in Fig. 7.

In HANGI and HANCGI, the algorithm starts by taking as input a decision table $<U,A,D>$, a neighborhood radius $\sigma$, and a minimum threshold $\beta$ for the relative importance of candidate features with respect to the reduced subset. Subsequently, the reduced subset and the candidate feature subset are initialized. An evaluation is made to determine if the candidate feature subset is empty. The current reduced subset is directly output if the candidate feature subset is empty. Conversely, if the candidate feature subset is not empty, all candidate features are iterated through. Each candidate feature’s importance concerning the reduced subset is computed using NGI or NCGI, denoted as $SIG(a_i,red,D)$. The feature with the highest importance is selected, and its importance is marked as $SI{G}_{max}$. Following this, an assessment is carried out to determine whether $SI{G}_{max}>\beta$. If true, the feature with the highest importance is removed from the candidate feature subset and incorporated into the reduced subset. The candidate feature subset is then revisited. Output the reduced subset until the candidate feature subset is empty or $SI{G}_{max}\le \beta$.

Assuming a dataset contains n samples, m features, and r categories, the best feature in each iteration is the one with the longest search time, with a worst-case search time of $(m^2+m)/2$. Calculate the time $n(n-1)/2$ required to determine the neighborhood relationship between samples in the dataset. The time to compute the Gini index within the neighborhood range is also nr. Therefore, the time complexity of the NGI and NCGI forward greedy feature selection algorithms is both $O(m^2{n}^2)$.

In HANGI and HANCGI, two parameters, $\sigma$ and $\beta$, are present. Parameter $\sigma$ controls the neighborhood radius, which determines the granularity of the neighborhood particles. The parameter $\beta$ is a threshold that stops the algorithm when the reduction of the GI is less than a particular value. Theoretically, the optimal values for these two parameters should be searched from the entire range of the dataset's space. Fortunately, as discussed in Hu et al. (2008, 2011), for algorithms with two parameters, such as the neighborhood rough set model, it is possible to approximate the optimal performance of the algorithm if one parameter is fixed at a particular value and the optimal value of the other parameter is searched across the entire space. Since the meaning of the same-sized evaluation metric in different algorithms is not the same, in this case, all $\beta$ values in all algorithms are set to 0. This means that adding a new feature will not lead to any improvement. Based on this, in the experimental analysis section, the value of parameter $\beta$ is set as a constant 0, and the optimal value for the neighborhood radius $\sigma$ is searched within the interval [0, 1], with a step size of 0.025.

Training parameters

In NRS-based models, the size of the neighborhood granule significantly impacts the model results. Determining the neighborhood granule’s size is essential to achieve optimal experimental outcomes. Thus, we employ ten-fold cross-validation with a step size of 0.025 in the range (0, 1) (Wang et al., 2019b) to obtain the optimal neighborhood radius parameter $\sigma$ for each algorithm. The search range in the “Spambase” dataset is (0, 0.225). Subsequently, we use four datasets and one algorithm to illustrate the selection process. Figure 8 displays the variation of classification accuracy with changing neighborhood radius for different datasets, using NGI as the evaluation metric.

Evidently, the neighborhood radius has a pronounced impact on classification accuracy. As the parameter changes, the four datasets exhibit varying accuracy levels in all classifiers. We select the radius that corresponds to relatively higher accuracy in all classifiers as the optimal radius. For instance, in the “Anneal” dataset, $\sigma =0.05$ is deemed the optimal neighborhood radius. Using the same training methodology, we determine the optimal neighborhood radius for each algorithm on various datasets, as presented in Table 2. In subsequent comparisons of algorithm performance, the neighborhood radius parameters are set based on this table.

In Table 2, the first column represents the dataset name, and each subsequent column header corresponds to the algorithm’s name. The values inside the table indicate the optimal neighborhood radius for each algorithm.

It is important to note that for the “Voting_records” dataset, HANRS cannot select features at any neighborhood radius. Therefore, we set it to the minimum value of 0.025 for subsequent comparisons.

Evaluation of feature validity

In the context of classification problems, feature selection algorithms aim to extract the most representative and discriminative features from the original feature set, creating a more compact subset. Constructing a classification model using the selected feature subset, achieving higher accuracy indicates that these features are more effective for the classification task on the given data. Based on the optimal neighborhood radius, the feature selection algorithms (HANRS, HANRE, HANDI, HANSI, HANGI, and HANCGI) were applied to 16 datasets, and the number of features selected is presented in Table 3. Where “Original” denotes the original dataset’s number of features, each subsequent column represents the average number of features selected by each algorithm over ten runs. Underscored numbers indicate the fewest selected features relative to other algorithms. Notably, HANRS did not select features in the “Voting_records” dataset and, therefore, is not included in the comparison.

Comparing the number of selected features in Table 3, we observe that HANGI and HANCGI successfully achieve feature reduction. There is no significant difference in the average number of features reduced among the six algorithms. HANMI shows the strongest reduction capability, while HANSI demonstrates the weakest. Across the 16 datasets, the average number of features was reduced by HANGI to 18, ranking fifth on average among the six algorithms. HANCGI reduces the average number of features to 11, ranking second on average among the six algorithms.

Next, we employ SVC, KNN, XGBoost, and ANN to train the selected feature subsets and compare their classification accuracies, as presented in Tables 4–7. Table 8 presents the mean accuracy of each dataset across the four classifiers. In these tables, underscored numbers indicate the best classification accuracy achieved through feature reduction relative to other algorithms.

Tables 4–8 demonstrate that the HANGI and HANCGI algorithms proposed in this article effectively improve classification accuracy. On average, HANGI improved classification accuracy on 14 datasets across the four classifiers, with improvements exceeding 10% on four datasets and an average accuracy improvement of 7%. HANCGI improved classification accuracy on 12 datasets, with improvements exceeding 10% on five datasets and an average accuracy improvement of 6.6%. Among the classifiers, XGBoost showed the smallest improvement in classification accuracy, with an average accuracy decrease of 0.8% for the selected features by HANCGI. This is because XGBoost not only acts as a classifier but also is an embedded feature selection model, automatically selecting features during the classification process to enhance accuracy. The results in Table 6 indicate that XGBoost’s feature selection results are similar to those of HANGI and HANCGI, with no significant difference in overall classification accuracy. In some datasets, the proposed algorithms have improved the classification accuracy of XGBoost by removing redundant and irrelevant features. For example, on the Toxicity dataset, both HANGI and HANCGI improved classification accuracy on XGBoost.

In the case of the 16 datasets, most of them showed improved classification accuracy after feature selection using the six feature selection algorithms. Although HANMI exhibited the strongest feature reduction capability, it had the poorest performance in terms of classification accuracy across the four classifiers. This suggests that HANMI might have discarded some crucial features during the feature selection process, resulting in lower classification accuracy.

HANDI, HANSI, the proposed HANGI, and HANCGI all showed relatively similar classification accuracy for the selected features across the four classifiers, but there were significant differences in some datasets. For example, on the DARWIN dataset, the features selected by HANCGI performed significantly better with SVC than HANDI, and the number of features selected by HANCGI was also much fewer than HANDI. On the “Parkinsons” dataset, despite a similar number of selected features between HANCGI and HANSI, HANCGI exhibited significantly lower classification accuracy with XGBoost.

It is worth mentioning that on the “Spambase” dataset, the selected features by all six feature selection algorithms achieved the same accuracy with XGBoost as the original dataset, even though the number of selected features differed among the algorithms. This suggests that all six feature selection algorithms got what XGBoost considered the crucial features, with HANDI selecting the fewest features and HANGI and HANCGI following closely behind.

Through comprehensive analysis of the experimental results, it is evident that the proposed methods often select fewer features while maintaining or improving classification accuracy. This suggests that the proposed methods can effectively eliminate more redundant attributes. Next, we evaluate the statistical significance of the performance differences among the six algorithms for feature selection through hypothesis testing. We first examine whether there are significant differences among the six algorithms on these datasets, utilizing the Friedman test statistic (Friedman, 1940):

(17) $${\chi }^2={\displaystyle \frac{12n}{k(k+1)}(\sum _{i=1}^k{r}_i^2-{\displaystyle \frac{k{(k+1)}^2}{4})}}$$

(18) $$F={\displaystyle \frac{(n-1){\chi }^2}{n(k-1)-{\chi }^2}}$$where $r_i$ represents the average rank of the algorithm, $n$ denotes the number of datasets, and $k$ represents the number of algorithms. The random variable “F” follows an F-distribution with degrees of freedom $k-1$ and $(k-1)(n-1)$. The critical value of the F-distribution at a significance level $\alpha$ can be obtained by invoking the subroutine ‘scipy.stats.f.ppf(1-α, n-1, (k-1)*(n-1))’ in Python 3.9. Thus, when $\alpha =0.05$, we obtain the critical value F(5,75) = 2.337. If the performance of the six algorithms is similar, the value of the Friedman statistic should not exceed the critical value F(5,75). Otherwise, there would be a significant difference in the feature selection performance among these six algorithms.

Table 9 displays the performance ranking order of the six algorithms’ selected features across the four classifiers arranged in ascending order. More significant numbers indicate better classification performance. According to the Friedman test statistic, we can obtain that F = 2.507 > 2.337 for the four classifiers. Evidently, there is a significant difference among the six algorithms on the four classifiers.

At this point, further post hoc tests are necessary to examine the differences among the six algorithms. The post hoc test employed here is the Nemenyi test. This statistical test requires determining the critical distance between average ranking values, defined by the following formula:

(19) $$C{D}_{\alpha }=q_{\alpha }\sqrt{{\displaystyle \frac{k(k+1)}{6N}}}$$where $q_{\alpha }$ is the critical tabulated value for this test. From Demšar (2006), we can obtain that $q_{0.05}=2.850$ when the number of algorithms is 6 and $\alpha =0.05$. It follows from the above formula that $C{D}_{0.05}=1.885(k=6,n=16)$. If the corresponding average rank distance is greater than the critical distance $C{D}_{0.05}$, it indicates a significant difference between the two algorithms.

It is easy to observe from Table 9 that the average ranking distance between HANSI and HANDI compared to HANMI is bigger than 1.885, indicating a significant difference in performance. However, the proposed HANGI and HANCGI, compared to the other four algorithms, have an average ranking distance of less than 1.885. This suggests that the algorithms proposed in the article do not exhibit a significant difference in average performance compared to the other four algorithms across the four classifiers.

Evaluation of algorithm stability

The stability of a feature selection algorithm refers to its ability to produce consistent or similar feature selection results when the dataset undergoes certain perturbations, such as removing or adding some samples. To discuss algorithm stability, we simulate removing a portion of samples. The procedure is as follows: First, the samples are randomly divided into ten subsets. In each iteration, nine subsets are chosen, and the feature selection algorithm is applied to obtain an optimal feature subset. This process is repeated ten times, resulting in 10 feature subsets. Features that appear at least five times out of the ten subsets are selected using a majority voting principle to form the final feature subset. Table 10 shows the number of features appearing at least once in each dataset’s ten feature selection results. Table 11 shows the number of features that appear at least five times in the ten feature selection results. Table 12 presents the ratio of feature numbers before and after voting, where a higher ratio indicates higher stability of the corresponding feature selection algorithm.

If a feature repeats occurrences across the ten feature subsets, it indicates a high level of reproducibility for that feature. The features that appear frequently in the ten subsets are selected through voting. If the final selected feature subset contains more features, it suggests a higher degree of algorithm stability. Table 12 presents the stability performance of the six algorithms across different datasets, with the most stable algorithm for each dataset marked with an underline.

Table 12 shows that HANSI has the highest stability, reaching 0.64, followed by HANCGI, with a stability of 0.59. The remaining algorithms have relatively similar stability, with HANMI showing the poorest stability. This difference in stability can be attributed to the fact that HANSI and HANCGI assess feature importance based on the class distribution within the class neighborhood range. However, other algorithms evaluate feature importance based on the class distribution within the sample neighborhood. When some samples are perturbed or removed, it inevitably affects the class distribution within their respective neighborhood range, thus influencing the assessment of feature importance.

Algorithms that assess feature importance by considering the distribution of classes within the sample neighborhood primarily focus on the local class distribution. When there is local perturbation, it directly impacts the evaluation of feature importance, resulting in lower stability. In contrast, HANCGI and HANSI pay more attention to the class neighborhood’s class distribution. When local interference occurs, it first affects the neighborhood of their respective classes, and in this process, interference is averaged out by unaffected samples within that class. Subsequently, feature importance assessment is influenced by the class neighborhood, and during this process, it is further averaged out by other unaffected classes. Therefore, these algorithms exhibit higher stability. HANCGI considers the distribution of all classes within the class neighborhood range, while HANSI only considers whether the classes within the class neighborhood are the same as the primary class. Therefore, HANSI exhibits stronger robustness to disturbances.

The experimental results above demonstrate that the algorithms proposed in this article exhibit high stability and strong feature reduction capabilities, particularly in removing redundant and irrelevant features, resulting in improved classification accuracy on most datasets. HANSI demonstrates the highest stability and achieves the best classification performance across the four classifiers, but it has the weakest feature reduction capability. On the other hand, HANGI, proposed in this article, has stronger feature reduction capabilities than HANSI, with slightly lower stability, and its selected features exhibit classification performance just 0.06% worse than HANSI on average. HANCGI boasts significantly stronger feature reduction capabilities than HANSI, with slightly less stability, and its selected features have an average classification performance of only 0.3% worse than HANSI.

In conclusion, compared to four classical feature selection algorithms based on neighborhood rough sets, the algorithms proposed in this article outperform three and have advantages and disadvantages compared to HANSI.