Approximate spectral clustering using both reference vectors and topology of the network generated by growing neural gas

Algorithm

ASC with GNG consists of generating a network by GNG, partitioning reference vectors by SC, and assigning data points into clusters. The algorithm for ASC with GNG is given in Algorithm 1.

Let us consider a set of N data points, X = { x₁, x₂,…,x_n,…,x_N}, where x_n ∈ R^d (Fig. 1A). The data point n is denoted by x_n. In this study, each data point is previously normalized by ${\mathbf{x}}_n\leftarrow {\mathbf{x}}_n/\parallel {\mathbf{x}}_{\mathrm{m}\mathrm{a}\mathrm{x}}\parallel$, where x_max is the data point that has the maximum norm. For an actual application, such normalization will not be required because we can use specific parameters for the application.

GNG generates a network from data points (Fig. 1B). The generated network is not a complete graph, and its topology reflects input space. The unit i in the network has the reference vector w_i ∈ R^d. The algorithm of GNG is denoted in Appendix.

The similarity matrix A ∈ R^{M × M} is calculated from the reference vectors and the topology of the generated network. M is the number of units in the network. If the unit i connects to the unit j, the element a_ij of A is defined as follows:

(1) $$a_{ij}=\exp (-{\displaystyle \frac{\parallel {\mathbf{w}}_i-{\mathbf{w}}_j{\parallel }^2}{2{\sigma }^2}}),$$

where w_i and w_j are the reference vectors of i and j, respectively. Otherwise, a_ij = 0. Equation (1) is the Gaussian similarity function that is most widely used to obtain a similarity matrix. The proposed method’s new point is to make a similarity matrix from both the reference vectors and the network topology. The similarity matrix is sparse because of using the network topology. However, the similarity matrix has information about significant connections representing input space.

The reference vectors are merged to k clusters by SC as shown in Fig. 1C. The normalized Laplacian matrix L_sym ∈ R^{M × M} is calculated from A. Here, we define the diagonal matrix D ∈ R^{M × M} to calculate the Laplacian matrix. The element of the diagonal matrix is $d_i={\sum }_{j=1}^M{a}_{ij}$. The Laplacian matrix is L = D − A. We derive the normalized Laplacian matrix from the following equation:

(2) $$L_{\mathrm{s}\mathrm{y}\mathrm{m}}=D^{-1/2}L{D}^{-1/2}.$$

Here, we calculate the k first eigenvectors u₁,…,u_k of L_sym. Let U ∈ R^{M× k} be the matrix containing the vector u₁,…,u_k as columns. For i = 1,…,M, let y_i ∈ R^k be the vector corresponding to the i-th row of U. k-means assigns (y_i)_{i = 1,…,M} to clusters C₁,…,C_k. Each row vector corresponds with each reference vector one-to-one.

Each data point is assigned to the cluster to which the nearest reference vector belongs. Finally, the data points are assigned to the clusters.

ASC with SOM and its alternatives

In this study, ASC with GNG is compared with ASCs with NG, SOM, k-means, and GNG using no topology to investigate the difference in clustering performances depending on the methods used to generate the network. ASCs with NG and SOM use NG and Kohonen’s SOM to generate networks instead of GNG, respectively. The weights of the network generated by NG and SOM are calculated using the Gaussian similarity function. ASC with k-means uses centroids generated by k-means as units in the network. The network is fully connected, and its weights are calculated using the Gaussian similarity function. ASC with GNG using no topology uses only the reference vectors generated by GNG. In ASC with GNG using no topology, the network is fully connected, and its weights are calculated using the Gaussian similarity function. In this study, ASC with GNG is also compared with SC. SC uses a fully-connected network, the Gaussian similarity function, a normalized Laplacian matrix L_sym = D ^{− 1/2}LD ^{− 1/2}, and k-means.

Complexity

The time complexity of SC is O(N³) (Izquierdo-Verdiguier et al., 2015; Taşdemir, Yalçin & Yildirim, 2015; Wang et al., 2013), where N is the number of data points. The complexity relies on the eigendecomposition of a Laplacian matrix. The eigendecomposition takes O(N³) time.

The time complexity of ASCs with GNG is O(MT + M³ + NM), where M and T are respectively the number of units and the number of iterations. O(MT) is the time complexity of GNG because O(M) is required to find the best match unit of a presented data point at every iteration. To decide the best match unit, we need to compute the distance between every reference vector and a presented data point and find the minimum distance. Thus, we calculate and compare M distances at every iteration. O(M³) is the time complexity of SC to partition the units. O(MN) is the time complexity to assign data points to clusters. To assign data points to clusters, we determine the nearest reference vector of every data point. Thus, we calculate M distances for each data point. For $N\gg M$, O(MN) dominates the computational time of ASCs with GNG.

If we use a sort algorithm, such as quicksort, to simultaneously find the best match unit and the second-best match unit every iteration, the time complexity of ASC with GNG is O(T M log M + M³ + NM). In this case, GNG sorts the distances between all the reference vectors and a presented data point to finds the best match and the second-best match units at every iteration. The sort algorithm takes O(M log M).

The time complexity of ASC with NG is O(TM log M + M³ + NM). The complexity of NG is O(TM log M). In NG, the distances between all the reference vectors and a presented data point are calculated at every iteration. The calculation of the distances takes O(M). NG sorts the distances to obtain the neighborhood ranking of the reference vectors at every iteration. The sort of M distances takes O(M log M) when we use quicksort. NG updates the weights of all the reference vectors at every iteration. NG takes O(M) to update the weighs. Thus, the time complexity of NG is O(T(M + M log M + M)) = O(TM log M). For $N\gg M$, O(MN) dominates the computational time of ASC with NG.

The time complexity of ASC with SOM is O(MT + M³ + NM). O(MT) is the time complexities of SOM because it requires finding the best match unit of a presented data point and updating all the reference vectors at every iteration. For $N\gg M$, O(MN) dominates the computational time of ASCs with SOM.

The time complexities of ASC with k-means is O(MNT_kmeans + M³ + NM), where T_kmeans is the number of iterations in k-means. O(MNT_kmeans) is the time complexities of k-means because the distances between M centroids and N data points are calculated at every iteration. It is difficult to estimate T_kmeans, but the convergence of k-means is fast (Bottou & Bengio, 1994) and T_kmeans will be enough smaller than N. For $N\gg M$, O(MNT_kmeans + NM) dominates the computational time of ASCs with k-means. In this case, the computational time approximately linearly increases with N because T_kmeans will be enough smaller than N.

The space complexity of SC is O(N²) (Mall, Langone & Suykens, 2013) because the memory space is required to store the N × N similarity matrix. The space complexity of ASCs with GNG, NG, SOM, and k-means is O(N + M²). O(N) is the memory space to store the data points. O(M²) is the memory space to store the similarity matrix of reference vectors. For $N\gg M$, the space complexity of the ASCs is O(N). Thus, the space complexities of the ASCs are much smaller than that of SC.

The time complexities and the space complexities of ASCs and SC are summarized in Table 1.

Parameters

All the parameters of the methods, except for T, M_max, M, and l, are found using grid search to maximize the mean of the purity scores for the datasets used in “Clustering Quality”. The parameters of ASC with GNG are T = 10⁵, λ = 250, ɛ₁ = 0.1, ɛ_n = 0.01, a_max = 75, α = 0.25, β = 0.99, σ = 0.25. The parameters of ASC with NG are T = 10⁵, λ_i = 1.0, λ_f = 0.01, ɛ_i = 0.5, ɛ_f = 0.005, a_maxi = 100, a_maxf = 300, and σ = 0.25. The Parameters of ASC with SOM are T = 10⁵, γ₀ = 0.05, σ₀ = 1.0, and σ = 0.5. The Parameter of ASC with k-means is σ = 0.1. The parameters of ASC with GNG no edge are T = 10⁵, λ = 350, ɛ₁ = 0.05, ɛ_n = 0.01, a_max = 100, α = 0.5, β = 0.999, σ = 0.5. The Parameter of ASC with k-means and SC is σ = 0.1. The meanings of the parameters are shown in “Algorithm” and Appendix.

Computational time

This subsection describes the computational time of ASCs with GNG, NG, SOM, and k-means and SC. For the measurement of the computational time, the dataset has five clusters and consists of 3-dimensional data points. This dataset is generated by datasets.make_blobs, which is the function of scikit-learn. The dataset is called Blobs in this study. In this experiment, I use the computer with two Xeon E5-2687W v4 CPUs and 64 GB of RAM and cluster the dataset using only one thread to not process in parallel.

Figure 2A shows the relationship between the computational time and the number of data points for 10² units. The computational time of ASC with k-means is shortest under about 5 × 10⁵ data points. The computational time of ASC with k-means linearly increases from about 10⁴ data points. ASCs with GNG, NG, and SOM show better computational performance than ASC with k-means beyond about 10⁶ data points. The computational time of ASCs with GNG, NG, and SOM does not significantly change under about 10⁵ data points. However, the computational time of ASCs with GNG, NG, and SOM linearly increases with the number of data points from about 10⁶ data points. Under 10⁵ data points, the computational time of ASCs with GNG, NG, and SOM will mainly depend on the computational cost to generate a network because O(MT) and O(TM log M) are more than O(M³) and O(MN). On the other hand, the linear increase of the computational time of ASCs with GNG, NG, and SOM will be dominantly derived from the computational cost to assign data points to units. The computational time of ASC with SOM is shorter than that of ASCs with GNG and NG under about 10⁶ data points because SOM does not have the process to change the network topology. From about 10⁷ data points, the computational time of ASC with SOM is not different from ASCs with GNG and NG because the computational cost to assign data points to units is dominant. Under 10⁶ data points, the computational time of ASC with GNG is shorter than ASC with NG. This difference occurs because the mean of growing M of ASC with GNG during learning is less than M of ASC with NG. The computational time of SC is not shown for more than 10⁴ data points because the computational time of spectral clustering is too long compared to the other methods. These results show that the ASCs outperform SC in terms of computational time for a large dataset. However, the results also show that ASC using topology is not effective for a small dataset.

Figure 2B shows the relationship between the computational time and the number of units for 10⁶ data points. In ASC with GNG, the number of units means the maximum of the number M_max. The computational time of all ASCs linearly increases with the number of units. The computational time of the ASCs with GNG, NG, and SOM increases more slowly than k-means because the complexity of k-means depends on not only the number of units but also the number of data points.

Clustering quality

To investigate the clustering performances of the ASC with GNG, NG, and SOM, three synthetic datasets and six real-world datasets are used. The synthetic datasets are Blobs, Circles, and Moons generated by datasets.make_blobs, datasets.make_circles, and datasets.make_moons that are functions of scikit-learn, respectively. Blobs are generated from three isotropic Gaussian distributions. The standard deviation and the means of each Gaussian are default values of the generating function. Blobs can be partition by a linear separation method such as k-means. Circles consists of two concentric circles. The noise and the scale parameters of the function generating Circles are set at 0.05 and 0.5, respectively. Moons includes two-moons shape distributed data points. The noise parameter of the function generating Moons is 0.05. Circles and Moons are typical synthetic datasets that can not be partitioned by a linear separation method. The real-world data are Iris, Wine, Spam, CNAE-9, Digits, and MNIST (Lecun et al., 1998). Iris, Wine, Spam, CNAE-9, and Digits are datasets found in the UCI Machine Learning Repository. In this study, Iris, Wine, Digits, and MNIST are obtained using scikit-learn. Iris and Wine are datasets frequently used to evaluate the performance of a clustering method. Spam and CNAE-9 are word datasets. In this study, three attributions of Spam: capital_run_length_average, capital_run_length_longest, and capital_run_length_total, are not used. Digits and MNIST are handwritten digits datasets. Table 2 shows the numbers of classes, data points, and attributions of the datasets.

To evaluate the clustering methods, we use the purity score. Purity is given by $\mathrm{P}\mathrm{u}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}=1/N{\sum }_{i=1}^k{max}_j{n}_i^j$, where N is the number of data points in a dataset, k is the number of clusters, and n^j_i is the number of data points that belong to the class j in the cluster i. When the purity score is 1, all data points belong to true clusters.

Table 3 shows the purities of ASCs with GNG, NG, SOM, k-means, and GNG using no topology and SC. ASC with GNG shows the best accurate clustering results for five datasets: Circles, Moons, Spam, CNAE-9, and Digits. For Blobs, ASC with GNG displays the second-best performance, but the difference of purity between ASC with GNG, NG, and GNG using no topology and SC is small. For MNIST, ASC with GNG shows the second-best performance. ASC with NG also shows relatively high purities for Blobs, Circles, Moons, and Digits. However, for Iris and Wine dataset, the clustering performances of ASCs with GNG and NG are worse than the others. The lower performance of ASCs with GNG and NG may be caused by too many units for the number of data points. Perhaps, there will be the optimal number of units to make a better similarity matrix. The performance of ASC with SOM is worse than the other methods for the datasets apart from Spam. The bad performance of ASC with SOM may be caused by the feature of SOM that is the tendency to have null units often. This feature of SOM is unsuitable for ASC using topology. ASC with k-means shows relatively high performance for Blobs, Circles, Moons, Iris, Wine, and MNIST. For MNIST, the performance of ASC with k-means is best. ASC with GNG using no topology shows the best performance for Blobs, Circles, and Wine. For Moons and Spam, ASC with GNG using no topology displays the second-best performance. For Blobs, Circles, Moons, Wine, Spam, CNAE-9, and Digits, the performance of ASC with GNG using no topology is better than that of ASC with k-means. This result suggests that GNG can quantize dataset better than k-means in many cases. For MNIST, SC cannot perform clustering because overflow occurs. These results suggest that the network topology effectively improves the performance of the clustering, and GNG will generate the same or better quantization result than k-means.

Figure 3 shows the relationship between purity and the number of units for Circles, Iris, and MNIST to investigate the dependence of clustering performance on the number of units. For Circles, the clustering performances of ASCs with GNG and NG vary in convex upward. This result suggests that the performances become worse for the larger or smaller number of units than the optimal number of units and become better for the around optimal number of units. For Iris, the performances of ASCs with GNG and NG are high under about 50 and 36 units, respectively. From about 50 and 36 units, these performances go down with the number of units. This result suggests that the number of units used in Table 3 is too large to obtain the optimal performances for Iris and Wine. For MNIST, the performances of ASCs with GNG and NG improve with the number of units. For all cases, the performances of ASCs with k-means and GNG using no topology more weakly depend on the number of units. This result suggests that ASC not using network topology displays peak performance less than ASCs using network topology in many cases but its performance does not strongly depend on the number of units.

A self-organizing map and its alternatives

Self-organizing map (SOM) and its alternatives, such as NG and GNG, are artificial neural networks using unsupervised learning. They convert data points to fewer weights (representative vectors) of units in a neural network and preserve the input topology as the topology of the neural network. Kohonen’s SOM (Kohonen, 1990) is a basic and typical SOM algorithm. Kohonen’s SOM has the features that the network topology is fixed into a lattice (Sun, Liu & Harada, 2017) and the number of units is constant. NG has been proposed by Martinetz & Schulten (1991) and can flexibly change the topology of its network. However, in NG, we have to preset the number of units in the network. GNG (Fritzke, 1994) achieves to flexibly change both the network topology and the number of units in the network according to an input dataset. GNG can find the topology of an input distribution (Garca-RodrGuez et al., 2012). GNG has been widely applied to clustering or topology learning such as extraction of two-dimensional outline of an image (Angelopoulou, Psarrou & García-Rodríguez, 2011; Angelopoulou et al., 2018), reconstruction of 3D models (Holdstein & Fischer, 2008), landmark extraction (Fatemizadeh, Lucas & Soltanian-Zadeh, 2003), object tracking (Frezza-Buet, 2008), and anomaly detection (Sun, Liu & Harada, 2017).

Growing neural gas

Growing neural gas (GNG) can generate a network from a given set of input data points. The network represents important topological relations in the data points using Hebb-like learning rule (Fritzke, 1994). A network generated by GNG consists of units and edges that are connections between units. In GNG, not only the number of the units but also the topology of the network can flexibly change according to input data points. The unit i has the reference vector w_i ∈ R^d and summed error E_i. The edges are not weighted and not directed. An edge has a variable called age to decide whether the edge is deleted.

Let us consider the set of N data points, X = { x₁, x₂,…,x_n,…,x_N}, where x_n ∈ R^d. The algorithm of the GNG to make a network from X is given by the following:

• 1. Start the network with only two units that are connected to each other. The reference vectors of the units set two data points randomly selected from X.

• 2. Select a data point x_n from the dataset at random.

• 3. Find the winning unit s₁ of x_n by

(10) $$s_1=\arg \underset{i}{min}\parallel {\mathbf{x}}_n-{\mathbf{w}}_i\parallel .$$

Simultaneously, find the second nearest unit s₂.

• 4. Add the squared distance between x_n and ${\mathbf{w}}_{s_1}$ to the summed error $E_{s_1}$:

(11) $$E_{s_1}\leftarrow E_{s_1}+\parallel {\mathbf{x}}_n-{\mathbf{w}}_{s_1}\parallel .$$

• 5. Move ${\mathbf{w}}_{s_1}$ toward x_n by fraction ${\epsilon }_{s_1}$ of the total distance:

(12) $${\mathbf{w}}_{s_1}\leftarrow {\mathbf{w}}_{s_1}+{\epsilon }_{s_1}({\mathbf{x}}_n-{\mathbf{w}}_{s_1}).$$

Also move the reference vectors of the all direct neighbor units s_n of s₁ toward x_n by the fraction ${\epsilon }_{s_n}$ of the total distance:

(13) $${\mathbf{w}}_{s_n}\leftarrow {\mathbf{w}}_{s_n}+{\epsilon }_{s_n}({\mathbf{x}}_n-{\mathbf{w}}_{s_n}).$$

• 6. If s₁ and s₂ are connected by an edge, set the age of this edge to zero. If s₁ and s₂ are not connected, create the edge connecting between these units.

• 7. Add one to the ages of all the edges emanating from s₁.

• 8. Remove the edges with their age larger than a_max. If this results in nodes having no emanating edges, remove them as well.

• 9. Every certain number λ of the input data point generated, insert a new unit as follows:

• Determine the unit q with the maximum summed error E_q.

• Find the node f with the largest error among the neighbors of q.

• Insert a new unit r halfway between q and f as follows:

(14) $${\mathbf{w}}_r=({\mathbf{w}}_q+{\mathbf{w}}_f)/2.$$

• The number of units has the limit M_s.

• Insert edges between r and q, and r and f. Remove the edge between q and f.

• Decrease the summed errors of q and f by multiplying them with a constant α. Initialize the summed error of r with the new summed error of q.

• 10. Decrease all summed errors by multiplying them with a constant β

• 11. If the number of iterations is not T, go to step 2.