Random k conditional nearest neighbor for high-dimensional data

Random k conditional nearest neighbor

The random k conditional nearest neighbor (RkCNN) method combines kCNN classifiers that are used in different random feature subsets. Among the q features, a feature subset of size m (1 ≤ m ≤ q) is created using simple random sampling without replacement from the data feature space $\mathcal{F}$. Each component kCNN model is then trained on one of these subsets.

Let Y be the output variable with L classes and y be an observed value of Y. Let X be the n × q feature matrix consisting of n input vectors each of length q. Consider a random feature subset ${\mathcal{F}}_j\subseteq \mathcal{F}$ where the elements of ${\mathcal{F}}_j$ are selected features. The matrix corresponding to the random subset ${\mathcal{F}}_j$ is denoted by

${\displaystyle {\mathbf{X}}_j^{\prime }=\left(\begin{array}{ccc}\hfill \hfill & \hfill {{\mathbf{x}}^{\prime }}_1^T\hfill & \hfill \hfill \\ \hfill \hfill & \hfill ⋮\hfill & \hfill \hfill \\ \hfill \hfill & \hfill {{\mathbf{x}}^{\prime }}_n^T\hfill & \hfill \hfill \\ \hfill \hfill \end{array}\right),}$

where ${\mathbf{x}}_i^{\prime }$ is the feature vector of the ith observation with m features selected for ${\mathcal{F}}_j$.

When predicting the class of a new instance, the estimated probabilities are calculated by applying kCNN to the feature subset. In particular, for a given query vector x, kCNN looks up the k-th nearest neighbor from each class and assigns the corresponding class probabilities (1)${\displaystyle {\hat{P}}_j\left(Y=c|\mathbf{x}\right)=\frac{{∥{\mathbf{x}}^{\prime },{\mathbf{x}}_{k|c}^{\prime }∥}_2^{-1}}{\sum _{l=1}^L{∥{\mathbf{x}}^{\prime },{\mathbf{x}}_{k|l}^{\prime }∥}_2^{-1}}}$ where ${∥\cdot ∥}_2$ is the L2-norm, and ${\mathbf{x}}_{k|c}^{\prime }$ represents the k-th nearest neighbor of x′ from class c.

Applying Eq. (1) to all h random feature subsets results in h sets of distinct class probability estimates. The proposed method aggregates these class probability estimations using weighted averaging, where the weights depend on the level of informativeness of the feature subsets for classification. If none of the features in a random subset plays an important role in classification, class probability estimates obtained from this subset space will add noisy information, potentially deteriorating the classification performance. Consequently, the proposed method assigns weights to estimated class probabilities from individual classifiers based on the informative level of their respective subset spaces. Specifically, we quantify the informativeness level for each feature subset using the separation score (for a detailed description of the separation score, please refer to ‘Model Parameters’) and compute the weights accordingly.

Given a set of separation scores S₁, …, S_h, we sort the separation score in descending order and denote them using order statistics, such as S₍₁₎, S₍₂₎…S_(h). For effective weighting, the RkCNN algorithm disregards results from noisy feature subset spaces with relatively low separation scores. The weights are then computed using the relative magnitudes of the remaining scores. Specifically, we only consider the class probability results from the r subsets that correspond to the top r separation scores, denoted as S₍₁₎, …, S_(r). Based on these r remaining scores, the weights for each classifier are computed as

${\displaystyle w_{\left(j\right)}=\frac{S_{\left(j\right)}}{\sum _{l=1}^r{S}_{\left(l\right)}}.}$ RkCNN estimates the aggregated probability ${\displaystyle \hat{P}\left(Y=c|\mathbf{x}\right)=\sum _{j=1}^r\left[{\hat{P}}_{\left(j\right)}\left(Y=c|\mathbf{x}\right)\cdot w_{\left(j\right)}\right]}$ where ${\hat{P}}_{\left(j\right)}$ is the estimated probability obtained in the feature subset corresponding to S_(j). For the classification task, RkCNN chooses the class with the greatest probability. That is, ${\displaystyle \hat{Y}=arg{max}_c\hat{P}\left(Y=c|\mathbf{x}\right).}$ Algorithm 1 summarises the procedure of RkCNN. We also illustrate the procedure of RkCNN in Fig. 1. The tuning parameters for RkCNN are discussed in ‘The number of nearest neighbors: k’.

 
 
__________________________________________________________________________________________________________ 
Algorithm 1 RkCNN 
  1:  for j = 1 to h do 
 2:       Sample m features from the feature space F, denoted as Fj, where Fj ⊆ 
  F 
 3:       Calculate the corresponding Separation Score Sj =  BV 
WV 
  4:  end for 
 5:  Sort {Sj}hj=1  in descending order, denoting the sorted values as S(j), i.e., 
     S(1) ≥ S(2) ≥⋅⋅⋅≥ S(h) 
  6:  Sort Fj by S(j), denoted as F(j) 
  7:  for j = 1 to r do 
 8:       Use the matrix of F(j), X′(j), and y to construct a kCNN model, denote 
     as kCNN(j) 
  9:       Calculate weights w(j) =     S(j) _____∑ 
r 
    l=1 S(l) 
10:       Calculate ˆ P(j)(Y = c|x), the predicted probability of class c by kCNN(j) 
11:  end for 
12:  Integrated    probability    for    class    c   is,     ˆ P(Y     =      c|x)      = 
     ∑r 
        j=1 [ 
       ˆ P(j)(Y = c|x) ⋅ w(j)] 
13:  Classify x to class c which maximise  ˆ P(Y = c|x) 
_________________________________________________________________________________________________________________    

Separation score

The proposed RkCNN method assigns different weights to individual feature subsets. Utilizing noisy feature subsets that contain irrelevant features does not enhance model performance, it is advisable to assign small weights to such subsets. If a subset consists of features irrelevant to the classification task, no pattern is expected between the set of selected features and the class label. Conversely, kCNN will perform well in a subset in which the classes are well-separated.

To quantify the level of discriminative information available in any given feature subset, we use the between-group and within-group variances. Specifically, the variance between groups is defined as the variance among the center points of each class.

Let ${\bar{\mathbf{x}}}_{\mathbf{c}}$ be the mean vector for the class c and $\bar{\mathbf{x}}$ be the overall mean vector. For a feature subset ${\mathcal{F}}_j$, the between-group variance (BV) is the variance of L center points ${\displaystyle \text{BV}={\sum }_{c=1}^L\frac{{∥{\bar{\mathbf{x}}}_c^{\prime }-{\bar{\mathbf{x}}}^{\prime }∥}_2}{L-1}.}$ The variance within groups is the mean of the variances in each class. Let x_ic be the ith observations in class c, where i_c = 1, …, N_c. The variance for class c is ${\displaystyle Va{r}_c={\sum }_{i_c=1}^{N_c}\frac{{∥{\mathbf{x}}_{i_c}^{\prime }-{\bar{\mathbf{x}}}_c^{\prime }∥}_2}{N_c-1}}$ where N_c is the number of data points in class c.

The variance within groups is the average of L variances ${\displaystyle \text{WV}=\sum _{c=1}^L\frac{Va{r}_c}{L}=\sum _{c=1}^L\sum _{i_c=1}^{N_c}\frac{{∥{\mathbf{x}}_{i_c}^{\prime }-{\bar{\mathbf{x}}}_c^{\prime }∥}_2}{\left(N_c-1\right)L}.}$ Then, the separation score for the subset is defined as ${\displaystyle S=\frac{\text{BV}}{\text{WV}}=\left[\sum _{i=c}^L\frac{{∥{\bar{\mathbf{x}}}_c-{\bar{\mathbf{x}}}^{\prime }∥}_2}{L-1}\right]/\left[\sum _{c=1}^L\sum _{i_c=1}^{N_c}\frac{{∥{\mathbf{x}}_{i_c}^{\prime }-{\bar{\mathbf{x}}}_c^{\prime }∥}_2}{\left(N_c-1\right)L}\right].}$ A small S value indicates a relatively low BV compared to WV, suggesting that the class means are close to the overall mean with larger within-class variances. In such cases, the class means are relatively close to the overall mean and each class has a relatively larger variance. Hence, the class probability estimated by kCNN will be barely meaningful. Conversely, a large S value suggests that the feature subset’s BV is relatively large in comparison to its WV. This indicates that the class means are distant from the overall mean, and the data is well-separable within this feature subspace. Such a feature subset is highly advantageous for utilizing kCNN, as it allows the classifier to effectively distinguish between classes.

Note that the computation of separation scores is model-free, as the separation score does not require the implementation of the kCNN. This implies that the computation can be quick once the following two matrices have been calculated. ${\displaystyle \mathbf{B}=\left(\begin{array}{ccc}\hfill \hfill & \hfill \hfill & \hfill \hfill \\ \hfill {\mathbf{b}}_1\hfill & \hfill \cdots \hfill & \hfill {\mathbf{b}}_L\hfill \\ \hfill \hfill & \hfill \hfill & \hfill \hfill \end{array}\right)=\left(\begin{array}{ccc}\hfill \hfill & \hfill \hfill & \hfill \hfill \\ \hfill \left({\bar{\mathbf{x}}}_1-\bar{\mathbf{x}}\right)\circ \left({\bar{\mathbf{x}}}_1-\bar{\mathbf{x}}\right)\hfill & \hfill \cdots \hfill & \hfill \left({\bar{\mathbf{x}}}_L-\bar{\mathbf{x}}\right)\circ \left({\bar{\mathbf{x}}}_L-\bar{\mathbf{x}}\right)\hfill \\ \hfill \hfill & \hfill \hfill & \hfill \hfill \end{array}\right)}$

${\displaystyle \mathbf{W}=\left(\begin{array}{ccc}\hfill \hfill & \hfill \hfill & \hfill \hfill \\ \hfill {\mathbf{w}}_1\hfill & \hfill \cdots \hfill & \hfill {\mathbf{w}}_N\hfill \\ \hfill \hfill & \hfill \hfill & \hfill \hfill \end{array}\right)=\left(\begin{array}{ccc}\hfill \hfill & \hfill \hfill & \hfill \hfill \\ \hfill \frac{\left({\mathbf{x}}_1-{\bar{\mathbf{x}}}_1\right)\circ \left({\mathbf{x}}_1-{\bar{\mathbf{x}}}_1\right)}{N_1-1}\hfill & \hfill \cdots \hfill & \hfill \frac{\left({\mathbf{x}}_N-{\bar{\mathbf{x}}}_L\right)\circ \left({\mathbf{x}}_N-{\bar{\mathbf{x}}}_L\right)}{N_L-1}\hfill \\ \hfill \hfill & \hfill \hfill & \hfill \hfill \end{array}\right)}$ where ∘ is the element-wise product, b_c and w_i represent the c-th and ith column vectors of B and W, respectively.

For a feature subset ${\mathcal{F}}_j$, let z_j = (z_1j, z_2j…, z_qj) be the indicator vector, where z_ij takes the value 1 if ith feature is in ${\mathcal{F}}_j$ and 0 otherwise. Then BV and WV can be obtained by (2)${\displaystyle \text{BV}=\frac{\sum _{c=1}^L\sqrt{{\mathbf{b}}_c^T{\mathbf{z}}_j}}{L-1}\text{and}\text{WV}=\frac{\sum _{i=1}^N\sqrt{{\mathbf{w}}_i^T{\mathbf{z}}_j}}{L}.}$

The result Eq. (2) implies that once matrices B and W are obtained, the computation of BV and WV (and thus the separation score) for individual feature subsets can be quick because no additional computations for individual vectors in B and W are needed.

While RkCNN generates h random subsets, the method only uses the subsets with the top r (≤h) separation scores. Assuming that the number of informative features follows a hypergeometric distribution, in the presence of many non-informative features, most subsets may contain only a few informative features. This implies that the distribution of the separation score will likely be positively skewed. Using the top r subsets helps to block the contribution of non-informative subsets.

Selecting the top r subsets is analogous to filter methods in feature selection. While filter methods select top-ranked individual features, RkCNN ranks random subsets based on their separation scores.

It is worth noting that the proposed separation score is related to Fisher’s linear discriminant analysis (LDA) (Fisher, 1936) which measures the level of separation using the variances between and within the classes. Fisher’s LDA projects the data points onto a one-dimensional space such that the projections have large between-group variance and small within-group variance. Although both Fisher’s LDA and our separation score use the between-/within-class variances, Fisher’s LDA is designed to identify the vector that maximizes the ratio of between-group variance to within-group variance. In contrast, we use the separation score to quantify the level of separability in random feature subsets.

Model parameters

In this section, we discuss the model parameters of RkCNN.

Data generation

Following the methodology outlined in Gul et al. (2018), the informative features of the two classes are generated with correlated and uncorrelated structures respectively. Suppose there are d (≤q) informative features in the dataset. For class A, the feature variables are generated from a multivariate normal distribution $\mathcal{N}\left({\mu }_A,\omega \Psi \right)$, where the mean vector μ_A is uniform with each element set to 2. Here, ω is a scaling factor that controls the overall variance of the features, modulating the extent of spread within class A. The variance–covariance matrix Ψ, a d × d matrix, is defined as ${\displaystyle \Psi =\left(\begin{array}{cccc}\hfill {\sigma }_{1,1}\hfill & \hfill {\sigma }_{1,2}\hfill & \hfill \cdots \hfill & \hfill {\sigma }_{1,d}\hfill \\ \hfill {\sigma }_{2,1}\hfill & \hfill {\sigma }_{2,2}\hfill & \hfill \cdots \hfill & \hfill {\sigma }_{2,d}\hfill \\ \hfill \cdots \hfill & \hfill \cdots \hfill & \hfill \cdots \hfill & \hfill \cdots \hfill \\ \hfill {\sigma }_{d,1}\hfill & \hfill {\sigma }_{d,2}\hfill & \hfill \cdots \hfill & \hfill {\sigma }_{d,d}\hfill \end{array}\right)}$ where ${\sigma }_{i,j}={\left(1/2\right)}^{\left|i-j\right|}$. Conversely, the feature variables for class B are generated independently from $\mathcal{N}\left({\mu }_B,{\mathcal{I}}_d\right)$, where the mean vector μ_B has value 1 on all elements and ${\mathcal{I}}_d$ is the d × d identity matrix, leading to uncorrelated features.

In each class, features are generated from identical distributions, ensuring that informativeness and separability are uniformly controlled. Then, non-informative features are generated from an independent standard normal distribution to complete the dataset.

Additionally, to explore scenarios involving multiple classes, we extend the data generation scheme to include a third class, C. Instances for class C are generated from $\mathcal{N}\left({\mu }_C,{\omega }_C{\mathcal{I}}_d\right)$, where the mean vector μ_C has value 3 on all elements. ω_C is the variance parameter specific to class C, controlling the dispersion of feature values in Class C and influencing the degree of overlap with other classes, thereby affecting the classification challenge.

In the remainder of this section, we conduct a comprehensive examination of the performance of the proposed method across various simulation settings. This involves varying the data size, the number of non-informative features, the number of classes, and the value of ω.

Simulation results and analysis

Separation score vs number of informative features

We evaluate the proposed separation score as an informativeness measure of feature subsets. The relationship between the separation score and the number of informative features is illustrated in Fig. 2. For each subplot within the figure, the simulation data contained 1,000 instances per class, comprising 20 informative features and 500 non-informative features as noisy factors. We analyzed the distribution of 800 separation scores.

The result indicated that the separation score for a feature subset increased as the subset contained more informative features. Notably, the most significant increase in score occurred when moving from zero to one informative feature, i.e., transitioning from a subset of purely noisy features to one containing an informative element. This pattern held true in both binary and multi-class scenarios, emphasizing the separation score as a potential proxy for the unknown count of informative features in practical applications.

Effect of k and m

Figure 3 illustrates the number and proportion of informative features within the r = 200 contributing subsets across varying values of m. The left panel showed a positive correlation between m and the number of informative features. As more features were included in a random subset, the likelihood of sampling informative features increased. Conversely, the right panel revealed a negative correlation between m and the proportion of the informative features. This indicated that larger subsets tended to be more dominated by noisy features, diluting the concentration of informative elements.

Figure 4 shows RkCNN’s classification accuracy across different values of m and k in the binary and multi-class problems. We varied ω and ω_C to adjust the extent of spread in the classes A and C, affecting their degree of overlap. When the classes were well-separated (i.e., ω = 1 for the binary and ω = ω_C = 1 for the multi-class problems), neither k nor m significantly affected RkCNN’s accuracy. However, as the class overlap increased with higher ω and ω_C, the performance of RkCNN became sensitive to the choice of k and m. The decreased accuracy at higher m values was attributed to the raised proportion of non-informative features, as depicted in the right panel of Fig. 3, which compromised the precision of class probability estimations. Notably, smaller values of k generally performed well regardless of m in the majority of cases. Our result regarding the relationship between k and the level of class overlap is also consistent with that of García, Mollineda & Sánchez (2008) who use the kNN classifier. Based on these findings, smaller values of k and m were recommended to optimize the classification performance of RkCNN, particularly under conditions of significant class overlap.

Effect of r and h at different noisy levels

Figure 5 shows the accuracy of RkCNN across different values of r and h under various noisy conditions, with the number of non-informative features ranging from 100 to 1,000. We fix k = 1 and $m=\lceil \sqrt{q}\rceil$ for this analysis.

The results demonstrated that increasing h, the total number of feature subsets, improved model accuracy, and the marginal gains in accuracy decreased as h became larger. This pattern suggested that although larger values of h led to better classification results, the benefits of adding more subsets grow smaller, eventually leading to a stabilization of performance at higher values of h.

When the number of non-informative features was relatively small, such as 100 or 200, the performance of RkCNN showed relative insensitivity to changes in h, maintaining stable accuracy across a broad range of h values. Conversely, in circumstances with 500 and especially 1,000 non-informative features, classification accuracy improved significantly as h increased. This was due to the larger pool of candidate subsets, which provided a higher probability of selecting subsets enriched with informative features. This trend highlighted that beyond a certain point, further increases in h did not substantially boost performance, as the accuracy plateaued.

The variation in r (200, 300, 400) exhibited similar patterns across all scenarios, suggesting that while a very large r is not necessary for achieving optimal performance, maintaining a sufficiently large r ensures efficient use of computational resources without redundant calculations. Overall, the analysis underscores the importance of carefully adjusting h and r according to the noise level in the data to optimize the accuracy of the RkCNN model.

Comparison

We compare RkCNN with several nearest-neighbor-based methods, including kNN, kCNN, and random k nearest neighbor (RkNN) and the ensemble of subset of k nearest neighbor (ESkNN). The misclassification rates under different numbers of non-informative features are presented in Table 1 and the corresponding average runtimes of ensemble methods are presented in Fig. 6. The tuning parameters were fixed at k = 3 and r = 200 through the comparison. For ESkNN, all other tuning parameters were set according to the recommendations in Gul et al. (2018). Given the diversity in feature numbers across datasets, we set the value of m for three ensemble methods associated with the feature size, specifically $m=⌈\sqrt{q}⌉$.

Table 1:

Misclassification rate for the methods on datasets with 20 informative features and different numbers of non-informative features.

Num of feature	kNN	kCNN	RkNN	ESkNN	RkCNN
20	0.070	0.047	0.103	0.208	0.083
20+50	0.088	0.070	0.055	0.190	0.042
20+100	0.180	0.190	0.123	0.165	0.057
20+200	0.135	0.125	0.178	0.188	0.047
20+500	0.285	0.280	0.350	0.293	0.073
20+1000	0.283	0.298	0.420	0.333	0.113

DOI: 10.7717/peerjcs.2497/table-1

From the results in Table 1, kNN and kCNN performed competitively when only informative features were presented as shown in the first row of the table. Under these conditions, kCNN exhibited the lowest error rate among the individual methods, and RkCNN showcased the lowest error rate among the ensemble methods. This was expected since the effectiveness of feature utilization is maximized in the absence of noise. When non-informative features were included, RkCNN outperformed all other methods in terms of classification error rate. The error rate increased as more non-informative features were introduced into the dataset, yet RkCNN remained robust against this increase in noise. Among the three ensemble methods compared, RkCNN consistently outperformed both kCNN and ESkNN in all scenarios.

Figure 6 presents the runtimes of RkNN, ESkNN and RkCNN at r = 200. For all ensemble methods, runtimes increased with the number of features. Among the methods, RkNN exhibited the fastest runtimes compared to ESkNN and RkCNN. The runtime of RkCNN was affected by the choice of the parameter h. When h = r, RkCNN’s runtime was comparable to that of RkNN. However, when h = 3r, RkCNN’s runtime increased due to the additional computations for separation scores.

The performance variations across different numbers of non-informative features illustrated RkCNN’s robustness, leading us to further investigate the impact of the spread parameter ω on classification accuracy. The misclassification rates for different values of ω are presented in Table 2, offering insights into how each method copes with varying levels of entropy or uncertainty in class distributions.

Table 2:

Misclassification rate for the methods on datasets with 20 informative features, varying numbers of non-informative features, and different values of ω.

	200 non-informative features
ωk	kNN		kCNN		RkNN		ESkNN		RkCNN
	1	3	1	3	1	3	1	3	1	3
1	0.188	0.135	0.188	0.125	0.175	0.178	0.188	0.188	0.068	0.055
3	0.267	0.272	0.267	0.255	0.188	0.233	0.235	0.215	0.113	0.157
5	0.295	0.303	0.295	0.310	0.115	0.188	0.205	0.198	0.095	0.162
10	0.305	0.340	0.305	0.353	0.080	0.100	0.090	0.178	0.035	0.125
15	0.320	0.348	0.320	0.370	0.042	0.065	0.073	0.140	0.020	0.093
	500 non-informative features
1	0.323	0.285	0.323	0.280	0.338	0.338	0.262	0.248	0.085	0.057
3	0.323	0.380	0.323	0.350	0.368	0.365	0.305	0.298	0.127	0.150
5	0.335	0.370	0.335	0.370	0.280	0.308	0.225	0.223	0.140	0.140
10	0.373	0.385	0.373	0.397	0.193	0.210	0.140	0.111	0.057	0.110
15	0.360	0.400	0.360	0.413	0.123	0.155	0.090	0.113	0.042	0.080
	1,000 non-informative features
1	0.405	0.283	0.405	0.298	0.377	0.428	0.305	0.248	0.162	0.127
3	0.382	0.360	0.382	0.308	0.402	0.430	0.360	0.300	0.195	0.215
5	0.387	0.382	0.387	0.370	0.363	0.392	0.275	0.250	0.165	0.190
10	0.363	0.395	0.363	0.443	0.285	0.333	0.170	0.167	0.125	0.135
15	0.373	0.392	0.373	0.433	0.235	0.280	0.095	0.118	0.070	0.105

DOI: 10.7717/peerjcs.2497/table-2

As ω increased, particularly to 3 or 5, the ensemble methods exhibited their highest misclassification rates, indicating significant uncertainty in class distributions. At high levels of class overlap, nearest-neighbor-based approaches tended to perform well at k = 1, which is consistent with the results of García, Mollineda & Sánchez (2008). However, an interesting trend emerged where further increases in ω led to better classification results, suggesting a threshold beyond which the clarity of class separability enhances the models’ performance. RkCNN consistently outperformed other methods in all simulated cases, demonstrating superior robustness to both increased numbers of non-informative features and varied ω values.

This pattern was particularly notable in environments with higher noise levels. When 500 and 1,000 non-informative features were present, RkCNN’s advantage became more pronounced, confirming its effectiveness in managing complex scenarios with significant class overlap and high noise levels. Although the error rates generally increased with more non-informative features, RkCNN’s performance remained comparatively stable. This stability underscored RkCNN’s potential as a reliable tool for applications where data conditions were challenging.

Throughout this section, the comparative analyses and simulation studies have consistently highlighted RkCNN’s performance under various testing conditions. From managing datasets with large amounts of non-informative features to coping with increased entropy in class distributions, RkCNN proved to be effective in high-dimensional data that contain noisy features.