The k conditional nearest neighbor algorithm for classification and class probability estimation

K conditional nearest neighbor

In multi-class classification, an instance with a feature vector x ∈ ℝ^q is associated with one of the possible classes c₁, …, c_L. We assume a set of training data containing N classified instances. For any x and a given k, we denote by x_k|i the kth nearest neighbor of class c_i (i = 1, …, L). Let d(x, x_k|i) = |x − x_k|i| be the (Euclidean) distance between x and x_k|i. Figure 1 illustrates this showing the distance between x and the second nearest neighbor (i.e., k = 2) of each class.

Consider a hypersphere with radius d(x, x_k|i) centered at x. By the definition of x_k|i, the hypersphere contains k instances of class c_i. We may approximate the local conditional density f(x|c_i) as (1)${\displaystyle \hat{f}\left(\mathbf{x}|c_i\right)=\frac{k}{N_i{V}_{k|i}}}$ where V_k|i is the volume of a hypersphere with radius d(x, x_k|i) centered at x and N_i represents the number of instances classified as class c_i. This approximation was also introduced in Fukunaga & Hostetler (1975). The approximation assumes that f(x|c_i) is nearly constant within the hypersphere of volume V_k|i when the radius d(x, x_k|i) is small. Using the prior $\hat{p}\left(c_i\right)\approx \frac{N_i}{N}$ where $N={\sum }_{i=1}^L{N}_i$ and Bayes theorem, the approximate posterior may be obtained as (2)${\displaystyle {\hat{p}}_k\left(c_i|\mathbf{x}\right)=\frac{\hat{p}\left(c_i\right)\hat{f}\left(\mathbf{x}|c_i\right)}{\hat{f}\left(\mathbf{x}\right)}=\frac{1}{\hat{f}\left(\mathbf{x}\right)}\frac{k}{N{V}_{k|i}}.}$ Because ${\sum }_{i=1}^L\hat{p}\left(c_i|\mathbf{x}\right)=1$, we have $\hat{f}\left(\mathbf{x}\right)={\sum }_{i=1}^L\frac{k}{N{V}_{k|i}}$. Then, ${\hat{p}}_k\left(c_i|\mathbf{x}\right)$ may be obtained as (3)${\displaystyle {\hat{p}}_k\left(c_i|\mathbf{x}\right)=\frac{\frac{k}{N{V}_{k|i}}}{\sum _{j=1}^L\frac{k}{N{V}_{k|j}}}=\frac{{d\left(\mathbf{x},{\mathbf{x}}_{k|i}\right)}^{-q}}{\sum _{j=1}^L{d\left(\mathbf{x},{\mathbf{x}}_{k|j}\right)}^{-q}}}$ since V_k|i ∝ d(x, x_k|i)^q. The class with the shortest distance among the L distances has the highest posterior.

The results in Eq. (3) are affected by the dimension of the feature space (q); the class probabilities converge to binary output (1 if the distance is smallest and 0 otherwise) as q increases. This implies the estimated class probabilities will be extreme in high-dimensional data, which is not desirable especially when the confidence of a prediction is required. Since smoothing parameters can improve predictive probability accuracy (e.g., LaPlace smoothing for the Naive Bayes algorithm (Mitchell, 1997), we introduce an optional tuning parameter r as follows: (4)${\displaystyle {\hat{p}}_k\left(c_i|\mathbf{x}\right)=\frac{{d\left(\mathbf{x},{\mathbf{x}}_{k|i}\right)}^{-q∕r}}{\sum _{j=1}^L{d\left(\mathbf{x},{\mathbf{x}}_{k|j}\right)}^{-q∕r}}}$ where r ≥ 1 controls the influence of the dimension of the feature space q. As r increases, each posterior converges to 1∕L. That is, increasing r smoothes the posterior estimates.

The k conditional nearest neighbor (kCNN) approach classifies x into the class with the largest estimated posterior probability. That is, class $\hat{c}$ is assigned to x if ${\displaystyle \hat{c}=\underset{i}{argmax}{\hat{p}}_k\left(c_i|\mathbf{x}\right).}$ The proposed classifier is equivalent to kNN when k = 1. We summarize the kCNN classifier in Algorithm 1.

Note that r affects the class probabilities but not the classification. We will show in ‘Ensemble of kCNN’ that the tuning parameter affects the classification of the ensemble of kCNN, which is presented in ‘Ensemble of kCNN’.

 
_________________________________________________________________________________________________ 
 Algorithm 1: The k conditional nearest neighbor algorithm 
_________________________________________________________________________________________________ 
      Input: A training data set D, a new instance vector x with dimension q, a positive inte 
      ger k, parameter r, a distance metric d 
      for i = 1 to L do 
         (a) From D, select xk|i, the kth nearest neighbor of x for class ci 
         (b) Calculate d(x,xk|i), the distance between x and xk|i 
      end for 
      for i = 1 to L do 
        Obtain ˆ pk(ci|x) ←  d(x,xk|i)−q∕r ____∑Lj=1 d(x,xk|j)−q∕r 
      end for 
      Classify x into ˆ c if ˆ c = argmax i ˆ pk(ci|x) 
__________________________________________________________________________________________________    

Figure 2 illustrates an example of a two-class classification problem. For a given k, the method calculates the distance between x and the kth nearest neighbor of each class. When k = 1 and k = 3, class c₂ has a larger posterior probability than c₁ as the corresponding distance is shorter. When k = 2, however, the posterior probability for class c₁ is greater.

Convergence of kCNN

The following theorem says that as the training data increase, kCNN converges to the optimal classifier, the Bayes classifier.

Theorem (convergence of kCNN): Consider a two-class problem with c₁ and c₂ where p(c₁) > 0 and p(c₂) > 0. Assume that f(x|c_i) (i = 1, 2) is continuous on ℝ^q. If the following conditions (a) k → ∞, and (b) $\frac{k}{{min}_i{N}_i}\to 0$ are satisfied, then for any x where f(x) > 0, kCNN with r = 1 converges in probability to the Bayes classifier.

Proof: Since kCNN makes predictions by approximate posteriors in Eq. (2), it is sufficient to show that ${\hat{p}}_k\left(c_i|\mathbf{x}\right)$ converges in probability to the true posterior.

We first consider the convergence of the prior estimate $\hat{p}\left(c_i\right)=N_i∕N$. Let c^(j) be the class of the jth training instance. The prior estimate may be described as $\hat{p}\left(c_i\right)=\frac{1}{N}{\sum }_{j=1}^{N}I\left(c^{\left(j\right)}=c_i\right)$ where I is the indicator function. Hence, by the weak law of large numbers, $\hat{p}\left(c_i\right)\underset{}{\overset{p}{\to }}$ ¹ p(c_i).

We next show that the approximation $\hat{f}\left(\mathbf{x}|c_i\right)$ in equation Eq. (1) converges in probability to the true conditional density function. Let $f_N\left(\mathbf{x}\right)=\frac{k}{NV}$ be an estimate of the density function f(x) where V is the volume of the hypersphere centered at x containing k training instances. In Loftsgaarden & Quesenberry (1965), it is showed that f_N(x) converges in probability to f(x) if k → ∞ and $\frac{k}{N}\to 0$ as N increases. We may apply this result to the convergence of the conditional density functions. By the second condition, both $\frac{k}{N_1}$ and $\frac{k}{N_2}$ converge to zero. Hence, $\hat{p}\left(\mathbf{x}|c_i\right)$ converges in probability to the true conditional density function f(x|c_i).

Since $\hat{p}\left(c_i\right)\underset{}{\overset{p}{\to }}p\left(c_i\right)$ and $\hat{f}\left(\mathbf{x}|c_i\right)\underset{}{\overset{p}{\to }}f\left(\mathbf{x}|c_i\right)$, ${\displaystyle \hat{f}\left(\mathbf{x}\right)=\sum _{i=1}^2\hat{p}\left(c_i\right)\hat{f}\left(\mathbf{x}|c_i\right)\underset{}{\overset{p}{\to }}\sum _{i=1}^{2}p\left(c_i\right)f\left(\mathbf{x}|c_i\right)=f\left(\mathbf{x}\right).}$ Hence, the approximate posterior in Eq. (2) converges in probability to the true posterior. This implies that kCNN converges in probability to the Bayes classifier. ■

The theorem implies that a choice of k needs to be subject to conditions (a) and (b) as the size of the data increases.

Time complexity of kCNN

The time complexity of kNN is O(Nq + Nk) (Zuo, Zhang & Wang, 2008) (O(Nq) for computing distances and O(Nk) for finding the k nearest neighbors and completing the classification). In the classification stage, kCNN (a) calculates the distances between the test instance to all training instances from each class, (b) identifies the kth nearest neighbor from each class, and (c) calculates posterior estimates by comparing the L distances and assigns the test instance to the class with the highest posterior estimate. Step (a) requires O(N₁q + ... + N_Lq) = O(Nq) multiplications. Step (b) requires O(N₁k + ... + N_Lk) = O(Nk) comparisons. Step (c) requires O(L) sum and comparison operations. Therefore, the time complexity for kCNN is O(Nq + Nk + L). In practice, the O(L) component is dominated by the other components, since L is usually much smaller than N. That is, the difference in the complexities between kNN and kCNN is small.

Ensemble of kCNN

The illustrative example in Fig. 2 shows that the classification is affected by the choice of k. Therefore, we propose an ensemble version of kCNN that combines the multiple kCNN algorithms with different values of k. Ensembles are well known as a method for improving predictive performance (Wu et al., 2008; Rokach, 2010). The ensemble of k conditional nearest neighbor (EkCNN) method makes a prediction based on the averaged posteriors for different values of k. These values are now indexed by w: w = 1, …, k. In the ensemble EkCNN, k represents the number of ensemble members. Suppose that posterior probability ${\hat{p}}_w\left(c_i|\mathbf{x}\right)$ is estimated by Eq. (4) for each w = 1, …, k. For a new instance x, the predicted class $\hat{c}$ is determined by ${\displaystyle \hat{c}=\underset{c_i}{argmax}\hat{p}\left(c_i|\mathbf{x}\right)=\underset{c_i}{argmax}\frac{1}{k}\sum _{w=1}^k{\hat{p}}_w\left(c_i|\mathbf{x}\right).}$ That is, EkCNN assigns x to the class with the highest average posterior estimate. Unlike kCNN that ignores the first k − 1 nearest neighbors of each class, EkCNN takes into consideration all k distances of each class. Using multiple values of k makes the prediction less reliant on a single k. This may improve the prediction result when the estimated class probabilities are highly variable as a function of k. An illustrative example in Fig. 3 shows that kCNN predicts either point A or point B incorrectly depending on the choice k = 1 or k = 2. However, EkCNN for k = 2 successfully predicts both A and B.

The complexity of EkCNN may be obtained analogously to steps (a)–(c) in ‘Time complexity of kCNN’. The complexities of EkCNN required in step (a) and (b) are the same as those of kCNN. In step (c), EkCNN requires O(kL) sum and comparison operations. Hence, the complexity of EkCNN is O(Nq + Nk + kL).

Data sets

We evaluated the proposed approaches using real benchmark data sets available at the UCI machine learning repository (Lichman, 2013). (We chose data sets to be diverse; all data sets we tried are shown.) Table 1 shows basic statistics of each data set including its numbers of classes and features. All data sets are available online at: https://archive.ics.uci.edu/ml/datasets.html. The data sets are ordered by the number of instances.

Experimental setup

We compared kCNN and EkCNN against kNN, WkNN, PNN, LMkNN and MLM-kHNN. Moreover, we considered an ensemble version of kNN (EkNN). EkNN estimates the probability for class c_i as ${\displaystyle \hat{p}\left(c_i|\mathbf{x}\right)=\frac{1}{k}\sum _{w=1}^k{\hat{p}}_w\left(c_i|\mathbf{x}\right)}$ where ${\hat{p}}_w\left(c_i|\mathbf{x}\right)$ is the probability estimated by kNN based on w nearest neighbors. Since MLM- kHNN is an ensemble of LMkNN using the harmonic mean, no additional ensemble model was considered. For EkCNN, we used r = q where q is the number of features of the data set. For kCNN and EkCNN, we added ϵ = 10⁻⁷ to each distance in equation Eq. (4) to avoid dividing by zero when the distance is zero.

For evaluation, we choose error rate (or equivalently accuracy), since error rate is one of the most commonly used metric and the skewness of the class distribution is not severe for most of the chosen data sets. The percentage of the majority class is less than 80% for most data sets (19 out of 20 data sets).

The analysis was conducted in R (R Core Team, 2014). For assessing the performance of the classifiers, we used 10-fold cross validation for each data. In the experiments, we varied the size of the neighborhood k from 1 to 15. For each method except PNN, the optimal value of k has to be determined based on the training data only. To that end, each training fold of the cross-validation (i.e., 90% of the data) was split into two random parts: internal training data (2/3) and internal validation data (1/3). The optimal k was the value that minimized classification error on the internal validation set. For PNN, Markov Chain Monte Carlo (MCMC) simulated samples are required. Following Holmes & Adams (2002), we used 5,000 burn-in samples, and retained every 100th sample in the next 50,000 samples.

We applied the Wilcoxon signed-rank test (Wilcoxon, 1945; Demšar, 2006) to carry out the pairwise comparisons of the methods over multiple data sets because unlike the t–test it does not make a distributional assumption. Also, the Wilcoxon test is more robust to outliers than the t-test (Demšar, 2006). The Wilcoxon test results report whether or not any two methods were ranked differently across data sets. Each test was one-sided at a significance level of 0.05.

Results

Table 2 summarizes the error rate (or misclassification rate) of each approach on each data set. Parameter k was tuned separately for each approach. EkCNN performed best on 8 out of the 20 data sets and kCNN performed best on 2 data sets. EkCNN achieved the lowest (i.e., best) average rank and kCNN the second lowest average rank. In the cases where kCNN performed the best, EkCNN was the second best method. According to the Wilcoxon test, EkCNN had a significantly lower (i.e., better) rank than kNN, EkNN, WkNN, LMkNN and kCNN with p-values less than 0.01. There was marginal evidence that EkCNN had a lower average rank than MLM- kHNN (p-value = 0.0656). Also, kCNN performed significantly better than kNN (p-value = 0.001), EkNN (p-value = 0.003), WkNN (p-value = 0.024), PNN (p-value = 0.003) and LMkNN (p-value = 0.041).

Equation (4) contains a tuning parameter r. As mentioned above, increasing r smoothes posterior estimates. For the results of EkCNN presented in Table 2, we chose r = q for all data sets. While not shown here, using r = q resulted in lower or equal error rates compared with using r = 1 on 18 out of 20 data sets. Specifying r = q reduced the error rate up to 6% relative to the error rate for r = 1.

Illustrating the choice of r on the sonar data set

We investigated the impact of r and ϵ on error rate of EkCNN (Classification by kCNN is affected by neither r nor ϵ.) for the sonar data set. Figure 4 shows that the error rate varied little for small values of k. For this data set, larger values of r are consistently preferable to smaller values. Note that error rates for r = 60 were almost identical to those for r = 100.

Decision boundary of kCNN and EkCNN with varying k

This section illustrates that the decision boundary between classes is smoother as k increases for both kCNN and EkCNN. We used a simulated data set from Friedman, Hastie & Tibshirani (2001). The classification problem contains two classes and two real valued features.

Figure 5 shows the decision boundary of kCNN with different k (solid curve) and the optimal Bayes decision boundary (dashed red curve). Increasing k resulted in smoother decision boundaries. However, when k is too large (e.g., k = 30 in this example), the decision boundary was overly smooth.

Analogously, Fig. 6 shows the decision boundary of EkCNN at r = 2 and different values k. Similar to kCNN, the decision boundary was smoothed as k increased. However, the magnitude of the changes was less variable. For example, the decision boundaries of EkCNN at k = 10 and k = 30 were similar, while those of kCNN were quite different.

Comparison of the posterior probability distribution of kNN and kCNN

Rather than considering classification, this section compares kCNN with kNN in terms of posterior probabilities. Probabilities are of interest, for example, when evaluating the entropy criterion. Using the same data set as in ‘Decision boundary of kCNN and EkCNN with varying k’, we plot the full posterior probability contours of kNN and kCNN in Fig. 7. We set r = q = 2 for kCNN. For k = 1, as expected, the posteriors estimated by kNN was always either 0 or 1. By contrast, kCNN provided less extreme posterior results even at k = 1. The posterior probabilities changed more gradually.

When k = 3, posterior probabilities from kNN jumped between four possible values (0, 1/3, 2/3, 1), whereas those from kCNN were much smoother. The result shows that unlike kNN, kCNN can produce smooth posterior probability fields even at small values of k.

Under what circumstances does the proposed method beat kNN?

kCNN (or EkCNN) may be useful when the true posterior distribution has a full range of probabilities rather than near dichotomous probabilities (close to 0 or 1). This occurs when the distributions of the classes substantially overlap. When the distribution of each class is well separated, for any data point the classification probabilities will be (near) 1 for one class and (near) 0 for the other classes. Otherwise, when the distributions overlap, the classification probabilities will be less extreme.

We conducted a small simulation to illustrate that kCNN is preferable to kNN when k is small and the distributions of the classes overlap. Assume that instances from each class are independently distributed following a multivariate normal distribution. Denote by μ_i the mean vector and by ∑_i the covariance matrix of class c_i. The parameters were given as ${\displaystyle {\mu }_1=\left(0,0,\dots ,0\right),{\sum }_1=I_q}$ ${\displaystyle {\mu }_2=\left(\frac{s}{\sqrt{q}},\dots ,\frac{s}{\sqrt{q}}\right),{\sum }_2=I_q}$ where I_q is the q dimensional identity matrix. Note that s is the Euclidean distance between the two means. Therefore, s controls the degree of overlap between the distributions of the two classes.

In order to obtain less variable results, we used 10 independent replicates for each parameter setting. The final outputs were obtained by averaging the results. We used 100 training and 1,000 test instances and the equal prior setting for the classes. Like Wu, Lin & Weng (2004), we evaluated the posterior estimates based on mean squared error (MSE). The MSE for the test data is obtained as ${\displaystyle MSE=\frac{1}{1000}\frac{1}{2}\sum _{j=1}^{1000}\sum _{i=1}^2{\left(\hat{p}\left(c_i|{\mathbf{x}}_j\right)-p\left(c_i|{\mathbf{x}}_j\right)\right)}^2}$ where x_j represents the jth test instance.

Table 3 shows the MSE for each method as a function of s and k when p = 2. The kCNN method beat kNN for small values of s. Small values of s imply that the mean vectors are close to each other, and hence there is more overlap between the two conditional densities. The difference in performance between the two methods decreased as s or k increased.

Next, we considered the effect of feature dimension q on each method. Table 4 shows the MSE for each method as a function of q and k when s = 0.1. Throughout the range of q, kCNN outperformed kNN. As q increased the MSE for kCNN was less affected by the choice of k.