MPV: a density-peak-based method for automated cluster number detection

Early algorithms (1973–2019)

Many researchers proposed a variety of algorithms from 1973 to 2019 (Maulik & Bandyopadhyay, 2002; Zhou & Xu, 2018; Masud et al., 2018; Gupta, Datta & Das, 2018; Fränti & Sieranoja, 2018; Khan et al., 2020). Due to space limitations, these early algorithms will not be detailed here. They laid the foundation for cluster number determination but often faced challenges with non-convex data structures.

Recent algorithms (2020–2024)

• In 2020, Rezaei & Fränti (2020) introduced using external indices to measure clustering stability, considering the cluster number optimal when results stabilize. Azhar et al. (2020) proposed the GMM-Tree algorithm, employing a hierarchical GMM model to partition datasets into GMM trees for determining cluster numbers and initial centers.

• In 2021, Guo et al. (2021) developed the FRCK algorithm, a fusion of rough clustering and K-means, showing good performance on optical datasets.

• In 2022, Rossbroich, Durieux & Wilderjans (2022) presented a chull algorithm based on ADPROCLUS for enumerating cluster numbers in overlapping datasets. Hsu & Phan-Anh-Huy (2022) proposed combining fuzzy C-means (FCM) and singular values to determine the optimal cluster number based on variance percentage. However, FCM’s limitation in handling non-convex datasets affects its effectiveness. Al-Khamees, Al-A’araji & Al-Shamery (2022) introduced an evolving Cauchy clustering algorithm, which determines the optimal cluster number through specific membership functions and evolving mechanisms. Yet, it is sensitive to parameters and struggles with non-spherical or non-convex datasets.

• In 2023, Mahmud et al. (2023) utilized the I-niceDP algorithm to identify cluster numbers and initial centers in large datasets, then combined random samples based on a spherical model. This method is effective for spherical datasets. Lippiello, Baccari & Bountzis (2023) proposed the similarity matrix sensitivity method, which determines cluster numbers from the block structure of the implicit similarity matrix. However, it is highly parameter-sensitive, risking over-merging or over-segmentation. Zhu & Li (2023) developed an algorithm using locally sensitive hashing for high-dimensional data, performing dimensionality reduction to determine cluster numbers. Nevertheless, hash function instability and potential data loss are concerns. Kasapis et al. (2023) proposed an algorithm for image datasets using an improved variance ratio standard. Fu, Jingyuan & Yun (2023) introduced the IMI2 algorithm, merging clusters based on separation values. Malinen & Fränti (2023) proposed optimizing clustering balance using All-Pairwise Distances, but it does not explicitly consider inter-cluster separation.

• In 2024, Lenssen & Schubert (2024) combined the silhouette idea with the PAM and FasterPAM algorithms to enhance the speed of cluster number determination.

Validity indices

Beyond algorithmic progress, the selection of validity indices is crucial for cluster number determination. Hassan et al. (2024) classified internal validity indices into four criteria: compactness, separability, exclusiveness, and incorrectness. Traditional indices like the Silhouette Index and Davies-Bouldin Index excel in spherical clusters but struggle with non-convexity due to their reliance on centroid distances. Recent proposals such as density-core-based indices (DCVI) and kernel density estimation (VIASCKDE) emphasize density peaks but often neglect inter-cluster boundary interactions. In contrast, our proposed MPV uniquely combines density peak separation (global perspective) and minimum inter-cluster distance (local perspective), effectively addressing both ring-shaped and high-density overlap scenarios.

Core principles

Density Peak Clustering (DPC), proposed by Rodriguez & Laio (2014), is a clustering algorithm that identifies cluster centers as data points with both high local density and large distances to other points of higher density. The key assumptions are:

(1) Cluster centers: Points with the highest local density ( ${\rho }_i$) and significant separation ( ${\delta }_i$) from other high-density points.

(2) Local density ( ${\mathit{\rho }}_{\mathit{i}}$): Calculated as the number of points within a predefined cutoff distance $d_c$:

$${\rho }_i=\sum _{j\ne i}\chi \left(d_{ij}-d_c\right),\chi \left(x\right)=\{\begin{array}{cc}1,& \hfill ifx<0\\ 0,& \hfill otherwise\end{array}.$$

(3) Relative distance ( ${\mathit{\delta }}_{\mathit{i}}$): The minimum distance to any point with higher density:

$${\delta }_i=\underset{j:{\rho }_j>{\rho }_i}{min}{d}_{ij}.$$

For the point with the highest density, ${\delta }_i=\underset{i\ne j}{max}{d}_{ij}$.

Algorithm workflow

(1) Compute pairwise distances: Construct the distance matrix $D=\left\{d_{ij}\right\}$.

(2) Calculate ${\mathit{\rho }}_{\mathit{i}}$ and ${\mathit{\delta }}_{\mathit{i}}$: For each point $z_i$, compute its local density ${\rho }_i$ and relative distance ${\delta }_i$.

(3) Decision graph: Plot ${\rho }_i$ vs. ${\delta }_i$ to visually determine the number of clusters.

(4) Cluster assignment: Assign non-center points to the cluster of their nearest higher-density neighbor.

Inter-cluster separation ( $\mathit{s}\mathit{d}$)

The separation degree sd integrates both global and local separation information:

(1) Density peak distance ( $\mathit{c}\mathit{e}\mathit{n}\mathit{d}$): Minimum distance between density peaks (cluster centers):

$$cend=\underset{{\mu }_i,{\mu }_j\in P}{min}\Vert {\mu }_i-{\mu }_j\Vert$$where $P=\left\{{\mu }_1,{\mu }_2,\dots ,{\mu }_3\right\}$ is the set of density peaks.

(2) Minimum inter-cluster distance ( $\mathit{c}\mathit{l}\mathit{d}$): Minimum distance between any two clusters:

$$cld=\underset{c_p,c_q}{min}\underset{z_i\in c_p,z_j\in c_q}{min}\Vert z_i-z_j\Vert .$$

(3) Balanced separation measure:

$$sd=\sqrt{cend\ast cld}.$$

This geometric mean ensures robustness against outliers while balancing global and local separation.

Intra-cluster compactness ( $\mathit{c}\mathit{d}$)

The compactness cd quantifies the tightness of the least compact cluster:

(1) Minimum intra-cluster distance: For cluster $C_k$, the minimum intra-cluster distance is calculated as:

$$unit\left(k\right)={\displaystyle \frac{1}{l_k-1}\sum _{i=1}^{l_k-1}\sum _{j=i}^{l_k}min\Vert z_i-z_j\Vert }$$where $l_k$ is the number of data points in cluster $C_k$.

Finding the minimum value among the data points here can better represent the compactness between points. Since the number of data points in each cluster is different, we cannot determine the compactness of a cluster solely based on the sum of distances. Therefore, calculating the average value can better demonstrate the compactness of the cluster.

(2) Global compactness:

$$cd=\underset{1<=k<=m}{max}unit\left(k\right).$$

This reflects the worst-case compactness to avoid underestimating loosely packed clusters.

It is necessary to clarify that the application of “average value” in this study differs essentially from that in traditional methods, resolving potential misunderstandings. Traditional methods (e.g., Calinski-Harabasz index) use the “average distance from all data points in a cluster to the centroid” to measure compactness. This centroid is a virtual point calculated artificially (not a real data point), leading to failure in ring or manifold datasets (consistent with the earlier statement that “average values perform poorly for non-spherical data”).

In contrast, the average value in MPV refers to the “average of the sum of minimum distances from each data point to its nearest neighbor within the cluster”—a calculation defined in the “Intra-Cluster Compactness” section. The computation is based on real pairwise distances between data points, and the final compactness cd is determined by taking the maximum of these average values across all clusters ( $cd=\underset{1<=k<=m}{max}unit\left(k\right)$). This design focuses on the “loosest cluster,” avoiding biases from virtual centroids and enabling effective compactness measurement for non-convex clusters (consistent with the statement that “average values better demonstrate cluster compactness”).

MPV formulation

The MPV dynamically balances separation and compactness:

$$MPV=\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{sd}{cd},1}\right).$$

Physical interpretation:

• Case 1: If $sd\ge cd$, $MPV={\displaystyle \frac{sd}{cd}}$, favoring clusters with strong separation.

• Case 2: If $sd<cd$, $MPV=1$, indicating insufficient separation to justify the current $K$.

Theoretical justification of distance-density synergy:

• (1)

  Necessity of the distance component ( $\mathit{s}\mathit{d}$):

  • If we only rely on the density component, the MPV cannot distinguish between clusters whose boundaries overlap but have significantly different densities. For example, in nested circular ring data, the distance between the density peaks of two clusters ( $cend$) may be large, but the minimum distance between clusters ( $cld$) approaches zero. At this time, the geometric mean $sd=\sqrt{cend\ast cld}$ approaches zero, and the MPV forcefully determines it as an invalid partition ( $MPV=1$), avoiding misjudgment.

  • Formula proof:

  • If $cld\to 0$ ⟹   $sd\to 0$⟹   $MPV=1$.

  • This indicates that the MPV suppresses the risk of over-segmentation that would occur if the analysis solely relied on density peaks through the distance component.

• (2)

  Necessity of the density component ( $\mathit{c}\mathit{d}$):

  • If we only rely on the distance component (such as the CH index), the MPV may overestimate the separability due to noise points. For example, noise points may increase the minimum distance between clusters ( $cld$), but at this time, the compactness ( $cd$) also increases synchronously due to the abnormal points within the cluster. Through the dynamic balance of $MPV=\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{sd}{cd},1}\right)$, the overestimation is avoided.

  • Formula proof:

  • If $sd$ increases abnormally, but $cd$ increases synchronously  ⟹   $MPV$ remains stable.

Compared to existing indices reviewed by Hassan et al. (2024), MPV diverges in three key aspects:

• (1)

  Density-distance synergy: Unlike CH or DB indices that solely use centroids, MPV integrates density peaks (real data points) and minimum cluster distances, avoiding artificial centroid biases in non-convex data.

• (2)

  Robust compactness definition: Traditional compactness metrics (e.g., Silhouette’s average intra-cluster distance) are sensitive to outliers. MPV uses the maximum intra-cluster minimum distance, prioritizing the least compact cluster to prevent underestimation.

• (3)

  Dynamic thresholding: While most indices rigidly maximize/minimize scores (e.g., Silhouette’s fixed [−1,1] range), MPV employs a self-adjusting ratio ${\displaystyle \frac{sd}{cd}}$, automatically suppressing over-segmentation when $sd<cd$.

Analysis of the MPV

In this section, we analyze MPV using the following legend. The similarity between two objects is inversely proportional to their distance: higher similarity corresponds to closer distance, while lower similarity corresponds to farther distance. Therefore, we employ Euclidean distance to assess both the compactness within clusters and the separation between clusters.

First, let’s discuss the process of determining the number of clusters using MPV on a spherical dataset. To better explain the advantages of MPV, we use the distribution of the dataset in Fig. 1 as an example. In Fig. 1, there are three clusters, among which ${\mu }_1$, ${\mu }_2$, and ${\mu }_3$ are the three density peaks, which are the cluster centers of the three clusters. As detailed in ‘Inter-cluster separation ( $sd$)’, the degree of separation is calculated by finding the minimum distance between density peaks (cend) and the minimum distance between clusters (cld). In the example shown in Fig. 1, $cend=min\{cend1,cend2,cend3\}$, $cld=min\{cld1,cld2,cld3\}$, Finally, the following can be obtained: $cend=cend3,cld=cld3$, so $sd=\sqrt{cen{d}_3\cdot cl{d}_3}$. The compactness of the cluster with ${\mu }_1$ as the center point is $unit\left(1\right)={\displaystyle \frac{e_1+e_2+e_3+e_4+e_5+e_6+e_7+e_8}{8}}$. Similarly, the intra-cluster compactness of the remaining two clusters can be computed. The maximum of these values is selected as the overall intra-cluster compactness, $cd=max\{unit(1),unit(2),unit(3)\}$.

Many algorithms use the average value of all data objects in a cluster as the center point to measure the degree of separation between clusters. While the average value can effectively represent a cluster in spherical datasets, it performs poorly for ring, semi-ring, and flow-shaped datasets. Because the object obtained by averaging is not within the cluster and is not a real data point, using it to measure inter-cluster separation can lead to inaccurate or erroneous results. To achieve a more balanced measure of inter-cluster separation, we utilize density peaks for calculation, as they represent real data points with high density. However, when the distance between clusters is very small, there is a high likelihood of a cluster being incorrectly split into two. Therefore, the minimum distance between clusters also significantly impacts the measurement of inter-cluster separation. Consequently, we employ sd to calculate the degree of separation between clusters.

To further validate the effectiveness of our method, we divided the dataset in Fig. 1 into four clusters, as shown in Fig. 2. We can observe that both the distance between density peaks ( $cen{d}_4$) and the minimum distance between clusters ( $cl{d}_4$) have suddenly decreased, while the compactness within the clusters remains almost unchanged. At this point, the average compactness within the clusters is similar to the value of $cl{d}_4$, indicating that the denominator of MPV Formulation has not changed significantly. However, the numerator suddenly shrank, resulting in a sharp decline in the MPV (Minimum Distance and Peak Variation). Therefore, the number of clusters is 3, not 4.

Next, we discuss the ring dataset and verify the effectiveness of our algorithm. At the same time, MPV performs well on ring, semi-ring, and flow-shaped datasets because it does not calculate the distance between all points and a center point when assessing cluster compactness. Instead, MPV starts from a point and identifies the closest neighboring point within the cluster. The distance to this nearest neighbor is recorded as the minimum distance from other points. In Fig. 3, there are two clusters, β and α. If ${\mu }_2$ is the center point of the cluster α, to calculate the distance from A to ${\mu }_2$ the compactness within the cluster cannot be expressed well. However, calculating the distance from A to B effectively indicates whether the cluster is compact. Additionally, the distance from A to C is much larger than that from A to B, which helps avoid misassigning point C to cluster α.

In Fig. 4, assuming that cluster α in Fig. 3 is divided into two clusters, we can observe that there are no significant changes in the compactness within the clusters or the values between density peaks. However, the inter-cluster distance $cl{d}_2$ suddenly decreases, which also causes a sudden drop in MPV. Consequently, the optimal number of clusters is determined to be 2 instead of 3.

Through the above arguments, we can see that MPV has a good effect on determining the number of clusters in the data.

Time complexity analysis

Assume that the data set has n objects and is divided into $k$ clusters. The time complexity for calculating $cend$ is $O(k2)$. The time complexity of calculating $cld$ is $O(n2)$. When computing the compactness within clusters, the time complexity is $O\left({\displaystyle \frac{n^2}{2}}\right)$, derived from the series $O(n-1+n-2+n-3+\dots 1)$. Consequently, the overall time complexity of MPV is $O(k2+n2+\frac{n^2}{2})=O(n2)$.

Use MPV to determine the best cluster number

In order to better reflect the intra-cluster compactness of all clusters, we select the maximum value of all intra-cluster compactness as the intra-cluster compactness of the dataset. MPV is a k-determination method that identifies the optimal $k$ by detecting the largest change in the separation measure $MPV\left(k\right)$. For the data set Parabolic5, the selection process of the best cluster number is shown in Fig. 5. Table 1 presents the detailed information of Parabolic5.

The formula for calculating the number of clusters is as follows:

$$K=\underset{2<=k<=kmax-1}{argmax}\left[MPV\left(k\right)-MPV\left(k-1\right)\right].$$

Here, $k$ represents the cluster number, $k_{max}$ represents the maximum value of the search range, and $K$ represents the optimal cluster number.

Experimental discussion on ring, semi-ring, and flow shape datasets

The true cluster number for the Parabolic2 dataset is 2. The results of the 14 methods used for obtaining the cluster number of Parabolic2 are shown in Figs. 11–14. From these figures, we can see that PC, I, CE, Dunn, CVDD, and our proposed method correctly identify the cluster number as 2. In contrast, the other seven methods fail to determine the correct cluster number. Specifically, ZXF yields a cluster number of 8; LL results in 3; LML and BIC both give 11; CH produces 10; Sil returns 6; and WB also indicates 10. The results from these methods are all incorrect.

Table 3 presents the experimental results on 21 artificial datasets, including spherical datasets such as S1–S4, A1–A3, and Unbalance. Figure 15 shows how many datasets each algorithm correctly determined the number of clusters for. FHV performs poorly, only identifying the correct number of clusters for three datasets with clear boundaries and minimal noise, which indicates its noise sensitivity.

LL, CH, and Dunn show better performance, correctly determining the number of clusters for over ten datasets. However, their effectiveness is mainly limited to spherical datasets, where they achieve an 87.5% accuracy rate. In contrast, their performance on ring-shaped and complex-shaped datasets is less ideal.

Sil and WB also perform well on spherical datasets, achieving a 100% accuracy rate. Yet, their performance on ring-shaped and complex-shaped datasets is not as satisfactory, with the highest accuracy rate peaking at 50%.

To further verify the robustness of MPV in scenarios unfavorable to density measurement, this section supplements the analysis of two types of typical complex datasets: the high-dimensional sparse dataset (Madelon, 500 dimensions) and the density-differentiated dataset (Unbalance).

1. High-dimensional sparse scenario (Madelon dataset): High-dimensional data often causes the failure of distance measurement in traditional distance-based or density-related internal validity indices due to the “curse of dimensionality” (e.g., Euclidean distance cannot distinguish inter-cluster differences). As shown in Table 3, MPV correctly identifies the optimal cluster number k = 2 for this dataset, while the Dunn index (a traditional distance-based index) and CVDD (a density-involved distance index) misjudge it as k = 51 and k = 3, respectively. This advantage stems from MPV’s design: it constructs the separation degree sd using “density peaks (real data points) + minimum inter-cluster distances”, replacing the “virtual centroid distance” relied on by traditional indices and avoiding the problem of centroids deviating from real cluster structures in high-dimensional spaces.

2. Density-differentiated scenario (Unbalance dataset): This dataset contains “low-density large clusters + high-density small clusters” (as shown in Table 1, the sample size is 6,500, and the real k = 8). Traditional methods tend to misclassify low-density clusters as noise due to the global density threshold. Experimental results show that MPV accurately identifies k = 8, while FHV misjudges it as k = 1 and LML as k = 21. This benefit comes from MPV’s “sd/cd dynamic balance mechanism”: when there is a large density difference between clusters, cd (compactness) captures the loose characteristics of low-density clusters through the “maximum intra-cluster average of minimum distances”, preventing sd (separation degree) from being over-amplified due to local high-density peaks. Thus, robust cluster number determination is achieved in density-differentiated scenarios.

The above results indicate that MPV can still maintain higher accuracy than traditional methods in high-dimensional and density-differentiated scenarios (which are unfavorable to density measurement), verifying that it is not only applicable to density-uniform scenarios such as ring and semi-ring datasets.

The proposed method in this article demonstrates excellent performance, correctly determining the number of clusters for 19 out of 21 datasets. It is particularly effective for ring-shaped datasets and achieves a 75% accuracy rate for spherical datasets.

Experimental discussion on real-world datasets

To further validate the effectiveness of our method, we applied 14 methods to estimate the cluster number on real-world datasets. Table 4 presents the experimental results on seven real-world datasets. Figure 16 shows the number of datasets for which each method correctly determined the number of clusters.

As indicated in Fig. 16, the performance of the Sil method on real-world datasets is inferior to that on artificial datasets. This is likely because the Sil method’s balance mechanism for compactness and separation is highly compatible with the geometric characteristics of spherical clusters, while real-world datasets often exhibit irregular structures. The PC, I, Dunn, and LL methods demonstrate relatively good performance on real-world datasets, correctly determining the number of clusters for at least five datasets. Notably, the method proposed in this article shows superior performance, accurately determining the number of clusters for all seven datasets. This fully proves the high effectiveness of the proposed method on real-world datasets.

Ablation study on MPV components

According to Table 5, we can draw the following experimental conclusions: