Probabilistically sampled and spectrally clustered plant species using phenotypic characteristics

Introduction

Genetic diversity has been an important foundation of plant breeding from the inception of agriculture since it helps develop new plants to meet the growing food demand globally. The breeding process is a complex combination of multiple stages (Louwaars, 2018). The first stage involves discovery of the native characteristics where the selection of diverse parent donors is of paramount importance (Swarup et al., 2020). One way plant genetic diversity can be studied is by using their phenotypic characteristics (physical characteristics). This kind of analysis can be relatively easily done because a sufficiently large amount of data is available from different geographical areas. In the phenotypic context, which is our first focus, a few characteristics that play an important role are Days to 50% Flowering, Days to Maturity, Plant Height, 100 Seed Weight, Seed Yield Per Plant, Number of Branches Per Plant, etc.

Cluster analysis is an important tool to describe and summarize the variation present between different plant species (Painkra et al., 2018). Thus, clustering can be used to obtain diverse parents which, as mentioned above, is of utmost importance. It is obvious that after clustering, the species present in the same cluster would have similar characteristics, while those present in different clusters would be diverse. Phenotypic data for the species of different plants (e.g., Soybean, Wheat, Rice, Maize, etc.) usually have enough variation for accurate clustering. However, if this data is obtained for the species of the same plant, then clustering becomes challenging due to less variation in the data, which forms our second focus.

Hierarchical Clustering (HC) is a traditional and standard method that is currently being used by plant biologists for grouping of phenotypic data (Painkra et al., 2018; Sharma et al., 2014; Kahraman, Onder & Ceyhan, 2014). However, this method has a few disadvantages. First, it does not provide the level of accuracy required for clustering similar species (Rokach, 2009). Second, HC is based upon building a hierarchical cluster tree (also called dendrogram), which becomes cumbersome and impractical to visualize when the data is too large. The second most common clustering algorithm that is being currently used widely is Unweighted Pair Group Method using Arithmetic Mean (UPGMA). This algorithm is a variant of HC, and hence, has the same two disadvantages as discussed above.

To overcome these two disadvantages, in this paper, we propose the use of the Spectral Clustering (SC) algorithm. SC is mathematically sound and is known to give one of the most accurate clustering results among the existing clustering algorithms (Luxburg, 2007). For genotypic data, we have recently shown substantial accuracy improvements by using SC as well (Shastri et al., 2019). Furthermore, unlike HC, SC does not generate the intermediate hierarchical cluster tree. To the best of our knowledge, this algorithm has not been applied to phenotypic data in any of the previous works (see the Literature Review section below).

HC, as well as SC, both are computationally expensive. They require substantial computational time when clustering large amounts of data (Luxburg, 2007; Mullner, 2013). Hence, we use sampling to reduce this complexity. Probability-based sampling techniques have recently gained a lot of attention because of their high accuracy at reduced cost (Tille, 2006). Among these, Pivotal Sampling is most commonly used, and hence, we apply it to phenotypic data (Chauvet, 2012). Like for SC, using Pivotal Sampling for phenotypic data is also new. Recently, Vector Quantization (VQ) has given promising results for genotypic data (Shastri et al., 2019). Hence, here we adapt VQ for phenotypic data as well. This also serves as a good standard against which we compare Pivotal Sampling.

To summarize, in this article, we develop a modified SC with Pivotal Sampling algorithm that is especially adapted for phenotypic data. The novelty of our work is in constructing the crucial similarity matrix for the clustering algorithm and defining the probabilities for the sampling technique. Although our algorithm can be applied to any plant species, we test it on around 2,400 Soybean species obtained from Indian Institute of Soybean Research, Indore, India (Gireesh et al., 2015). In the experiments, we perform four sets of comparisons. First, we show that use of Pivotal Sampling does not deteriorate the cluster quality. Second, our algorithm outperforms all the proposed competitive clustering algorithms with sampling in terms of the accuracy (i.e. modified SC with VQ, HC with Pivotal Sampling, and HC with VQ). The computational complexities of all these algorithms are similar because of the involved sampling. Third, our modified SC with Pivotal Sampling doubly outperforms HC, which as earlier, is a standard in the plant studies domain. In terms of the accuracy, we are up to 45% more accurate. In terms of complexity, our algorithm is more than a magnitude cheaper than HC. Fourth and finally, we demonstrate the superiority of our algorithm by comparing it with two previous works that are closest to ours.

The rest of this article is organized as follows. Section “Literature Review” provides a brief summary of the previous studies on phenotypic data. The standard algorithms for Pivotal Sampling and SC are discussed in Section “Sampling and Clustering Algorithms”. Section “Implementing Pivotal Sampling and Modified Spectral Clustering for Phenotypic Data” describes the crucial adaptations done in Pivotal Sampling and SC for phenotypic data. The data description, validation metric, and the experimental set-up are presented in Section “Methodology”. Section “Results and Discussion” gives the experimental results. Finally, conclusions and future work are provided in Section “Conclusions and Future Work”.

Literature Review

In this section, we present some relevant previous studies on phenotypic data and the novelty of our approach. Broadly, these studies can be classified into two categories. The first category consists of the works that identify relationships between the different phenotypic characteristics (for example, lower plant height may relate to lower plant yield or vice versa). These works are discussed in Section “First Category Previous Studies”. The second category consists of the studies that identify the species with dissimilar phenotypic characteristics for the breeding program. These studies are discussed in Section “Second Category Previous Studies”. Finally, we present a set of works that belong to both the categories in Section “Both Categories Previous Studies”.

Sampling and Clustering Algorithms

In this section, we briefly discuss the standard algorithms for Pivotal Sampling and SC in the two subsections below.

Implementing Pivotal Sampling and Modified Spectral Clustering for Phenotypic Data

Here, we first present the application of Pivotal Sampling to obtain the samples from phenotypic data. Subsequently, we implement our modified SC algorithm on the same data. Consider that the phenotypic data of a plant consists of n species with each specie evaluated for m different characteristics/traits. These characteristics may have categorical (non-numerical) or numerical values. Hence, we need to convert the categorical values into numerical ones. For this, we use the label encoder method (Hancock & Khoshgoftaar, 2020). This method transforms non-numerical labels into numerical values between 0 and (number of categories) – 1. For example, if a characteristic has three possible labels; poor, good, and very good, we use 0, 1, and 2 to represent them, respectively.

As discussed in Section “Pivotal Sampling”, Pivotal Sampling requires that the inclusion probabilities (i.e. π_i for i = 1,…, n), of all the species in the population U, be computed before a unit is considered for a contest. The set of characteristics associated with a specie can be exploited in computing these probabilities. To select a sample of size N, where $N\ll n$, we obtain these probabilities as Deville & Tille (1998)

(13) $${\pi }_i=N{\displaystyle \frac{{\varkappa }_i}{{\sum }_{i\in U}{\varkappa }_i},}$$where ϰ_i can be a property associated with any one characteristic (or a combination of them) of the i^th specie. Obtaining π_i in such a way also ensures that ${\sum }_{i=1}^n{\pi }_i=N$, i.e. we get exactly N selection steps, and in-turn, exactly N samples.

In our implementation, we use the deviation property of the species, which is discussed next. Since different characteristics have values in different ranges, we start by normalizing them as below (Jain, Nandakumar & Ross, 2005; Shastri, Tamrakar & Ahuja, 2018).

(14) $${\left({𝒳}_j\right)}_i=\frac{{(x_j)}_i-\text{min}(x_j)}{\text{max}(x_j)-\text{min}(x_j)}.$$

Here, ${\left({𝒳}_j\right)}_i$ and (x_j)_i are the normalized value and the actual value of the j^th characteristic for the i^th specie, respectively with j = 1,…, m and i = 1,…, n. Furthermore, max(x_j) and min(x_j) are the maximum and the minimum values of the j^th characteristic among all the species. Now, the deviation for the i^th specie is calculated using the above normalized values as

(15) $$de{v}_i={\displaystyle \sum _{j=1}^m\text{max}}\left({𝒳}_j\right)-{\left({𝒳}_j\right)}_i.$$

Here, $\text{max }\left({𝒳}_j\right)$ denotes the maximum normalized value of the j^th characteristic among all the species. Practically, a relatively large value of dev_i indicates that the i^th specie is less important, and hence, its probability should be small. Thus, the inclusion probability of a specie is calculated by taking ${\varkappa }_i=\frac{1}{de{v}_i}$ in Eq. (13) or

(16) $${\pi }_i=N{\displaystyle \frac{{\displaystyle \frac{1}{de{v}_i}}}{{\sum }_{i\in U}{\displaystyle \frac{1}{de{v}_i}}}.}$$

Thus, if the sum of probabilities of two species under consideration is greater than or equal to 1, we follow the selection step as discussed in Section “Pivotal Sampling”. On the other hand, we follow rejection step when this sum is less than 1. This process is repeated until we obtain N species.

Next, we discuss the clustering of these N species into k clusters. Similar to the standard SC algorithm discussed in Section “Spectral Clustering”, the first step in our modified SC is to obtain the similarity matrix. As mentioned earlier, this is the most important aspect of this algorithm since the better the matrix quality, the better the clustering accuracy. For this, we consider these N species as the vertices of a graph. Let vector p_i contain the normalized values of all the characteristics (m) for the i^th specie. Thus, we have N such vectors corresponding to the N species selected using Pivotal Sampling. That is, $p_i={[{\left({𝒳}_1\right)}_i,\dots ,{\left({𝒳}_m\right)}_i]}^T$ for i = 1,…, N. In our implementation, we use a fully connected graph to build the similarity matrix, i.e. we obtain similarities among all the N species.

We define the similarity between the vectors p1 and p2 (without loss of generality, representing the species 1 and 2, respectively) as the inverse of the distance between these vectors obtained by using the distance measures mentioned in “Spectral Clustering”. This is intuitive because smaller the distance between any two species, larger the similarity between them and vice versa. We denote this distance by d_p1p2. We build this matrix of size N × N by obtaining the similarities among all the N species.

The next step is to compute the Laplacian matrix, which when obtained from the above-discussed similarity matrix, generates poor eigenvalues,² and in-turn poor corresponding eigenvectors that are required for clustering³ . Thus, instead of taking only the inverse of d_p1p2, we also take its exponent, i.e. we define the similarity between the species 1 and 2 as $e^{-d_{p1p2}}$ (Ng, Jordan & Weiss, 2002; Nemade et al., 2018). This, besides fixing the poor eigenvalues/eigenvectors problem, also helps perform better clustering of the given data. Further, we follow the remaining steps as discussed in “Spectral Clustering”.

Above, we discussed the clustering of N sampled species into k clusters. However, our goal is to cluster all n species and not just N. Hence, there is a need to reverse-map the remaining n − N species to these k clusters. For this, we define the notion of average similarity, which between the non-clustered specie $\tilde{p}$ and the cluster C_l is given as

(17) $$𝒜𝓢(C_l,\tilde{p})={\displaystyle \frac{1}{\mathrm{\#}(C_l)}\sum _{q\in C_l}{e}^{-d_{\tilde{p}q}}.}$$

Here, #(C_l) denotes the number of species present in C_l and q is a specie originally clustered in C_l by our modified SC algorithm with Pivotal Sampling. We obtain the average similarity of $\tilde{p}$ with all the k clusters (i.e. with C_l for l = 1,…, k), and associate it with the cluster with which $\tilde{p}$ has the maximum similarity.

Next, we perform the complexity analysis of our algorithm. Since Pivotal Sampling and SC form the bases of our algorithm, we discuss the complexities of these algorithms before ours.

• 1. Pivotal Sampling (n: number of species, N: sample size)

• (a) Obtaining Samples: $𝒪(n)$

• 2. SC (n, m: number of characteristics)

• (a) Constructing Similarity Matrix: $𝒪(n^{2}m)$

• (b) Obtaining Laplacian Matrix: $𝒪(n^3)$

• 3. Our Algorithm (n, N, m)

• (a) Obtaining Samples: $𝒪(n)$

• (b) Constructing Similarity Matrix: $𝒪(N^{2}m)$

• (c) Obtaining Laplacian Matrix: $𝒪(N^3)$

• (d) Reverse Mapping: $𝒪((n-N)N)$

Thus, the overall complexity of our algorithm is $𝒪(nN+N^3+N^{2}m)$. Here, we have kept three terms because any of these can dominate (here, $n\gg N$, m).

When we compare complexity of our algorithm with that of HC, which is $𝒪(n^3)$, it is evident that we are more than a magnitude faster than HC. We revisit this complexity analysis after discussing data in the next section, which supports our claim further.

Methodology

In this section, we first briefly discuss the data used for our experiments. Next, we check the goodness of our sampling technique by estimating a measure called the population total. The hypothesis related to this is as follows: for a particular sampling technique, if the estimate (or approximation) of the population total using the samples is close to the actual population total, then that sampling technique is considered good in an absolute sense. Finally, we describe the clustering set-up, where the below are analyzed.

1. The Validation Metric. It is hypothesized that a good clustering is one where clusters are compact and well-separated.

2. The Ideal Number of Clusters. The hypothesis related to this is as follows: given a set of eigenvalues of the Laplacian matrix, we can exploit the differences between these eigenvalues to obtain the ideal number of clusters.

3. The Suitable Distance Measures. For building the similarity matrix and the Laplacian matrix, it is hypothesized to chose those matrices that give the best value for the validation metric.

Clustering setup

Here, first, we describe the criteria used to check the goodness of the generated clusters. There are two categories of metrics available for the validation of clustering algorithms. One category includes the metrics that require prior knowledge of the cluster labels (Fahad et al., 2014). On the other hand, metrics from the second category do not have this requirement (Fahad et al., 2014; Rousseeuw, 1987). In this work, the ideal cluster labels are not available, and hence, we use a metric called Silhouette Value (from the second category) for validation of our clustering algorithms (Rousseeuw, 1987).

Clustering is considered good if the obtained clusters are compact and well-separated. Silhouette Value captures both these aspects well by computing the intra-cluster similarity and the inter-cluster similarity. Consider that we have k clusters represented as C₁,…, C_k, and we want to obtain the Silhouette Value of the i^th data point present in the cluster C₁. For this, we compute the average distance between this data point and all the other points in the cluster C₁. This distance is denoted as a(i). Next, we compute the average distance between the i^th data point and all the other points in clusters C₂,…, C_k. This distance is denoted as b(i). Then, for this point, Silhouette Value is computed as below (Rousseeuw, 1987).

(21) $$s(i)={\displaystyle \frac{b(i)-a(i)}{max\{a(i),b(i)\}}.}$$

As evident from the above discussion, the intra-cluster similarity is captured by a(i), and the inter-cluster similarity is captured by b(i). This value usually lies between minus one to plus one because the denominator of Eq. (21) is always greater than its numerator. Silhouette Value for the overall clustering is obtained by averaging the Silhouette Values of all the data points. If this value tends towards a positive one, then the clustering is considered to be good. On the other hand, if this value tends towards a negative one, then the clustering is considered poor.

Second, we determine the ideal number of clusters by using the eigenvalue gap heuristic (Luxburg, 2007; Kong et al., 2010). If λ₁, λ₂, …, λ_n are the eigenvalues of the matrix used for clustering (e.g., the Laplacian matrix), then often the initial set of eigenvalues, say k, have a considerable difference between the consecutive ones in this set. That is, $|{\lambda }_i-{\lambda }_{i+1}|\text{⧸}\approx 0$ for i = 1,…, k − 1. After the k^th eigenvalue, this difference is usually approximately zero. According to this heuristic, this k gives a good estimate of the ideal number of clusters.

For this experiment, without loss of generality, we build the similarity matrix using the Euclidean distance measure on the above discussed phenotypic data. As mentioned earlier, it is recommended to use the Type-3 Laplacian matrix (Luxburg, 2007). Hence, we use its eigenvalues for estimating k. Figure 1 represents the graph of the first fifty smallest eigenvalues (in absolute terms) of this Laplacian matrix. On the x-axis, we have the eigenvalue number, and on the y-axis its corresponding value.

From this figure, we can see that there is a considerable difference between the first ten consecutive eigenvalues. After the tenth eigenvalue, this difference is very small (tending to zero). Hence, based upon the earlier argument and this plot, we take k as ten. To corroborate this choice more, we experiment with k as twenty and thirty as well. As expected, and discussed in-detail later in this section, Silhouette Values for these numbers of clusters are substantially lower than those for ten clusters.

Third, and final, we perform experiments to identify the suitable similarity measures to build the similarity matrix, and also verify that, as recommended, the Type-3 Laplacian matrix is the best. Table 6 below gives Silhouette Values of our modified SC for all seven similarity measures and three Laplacians when clustering the earlier presented phenotypic data into 10, 20, and 30 clusters.

Table 6:

Silhouette values for modified SC with seven similarity measures and three Laplacian matrices for k = 10, 20, and 30.

Silhouette values in bold represent good clustering.

Sr. No.	Similarity measure	Number of clusters (k)	Type-1 Laplacian	Type-2 Laplacian	Type-3 Laplacian
1.	Euclidean	10	0.0828	−0.0273	0.2422
		20	0.0455	−0.1096	0.2069
		30	0.0887	−0.1536	0.1783
2.	Squared Euclidean	10	0.0815	−0.0555	0.3836
		20	−0.0315	−0.1809	0.2612
		30	0.0354	−0.2367	0.1538
3.	City-block	10	0.0687	0.2375	0.2647
		20	−0.0356	0.1347	0.2082
		30	−0.0870	0.0866	0.1887
4.	Cosine	10	0.1737	−0.1408	0.0694
		20	0.0359	−0.1973	0.0277
		30	0.0245	−0.2456	−0.0316
5.	Correlation	10	0.1926	−0.1259	0.3426
		20	0.0970	−0.2198	0.2313
		30	0.2383	−0.2604	0.1556
6.	Hamming	10	0.0643	0.0706	0.0775
		20	0.0683	0.0311	0.0382
		30	0.0715	0.0283	0.0229
7.	Jaccard	10	0.0716	0.0303	0.0458
		20	0.0446	0.0276	0.0236
		30	0.0279	0.0298	0.0318

DOI: 10.7717/peerj-11927/table-6

From this table, it is evident that Silhouette Values for the Euclidean, Squared Euclidean, City-block and Correlation similarity measures and the Type-3 Laplacian matrix are the best. Hence, we use these four similarity measures and this Laplacian matrix. Also, as mentioned earlier, Silhouette Values decrease for twenty and thirty cluster sizes.

Results and Discussion

Using the earlier presented dataset, and sampling-clustering setups, we compare our proposed algorithm (i.e. modified Spectral Clustering (SC) with Pivotal Sampling) with the existing variants in four ways. Again, as earlier, we use Silhouette Values for comparison. Quantifying statistical difference between different Silhouette Values is a hard task. In general, the more closer these values are to one, the better is the clustering (see Section “Clustering Setup”).

First, we demonstrate that use of sampling with modified SC does not deteriorate the quality of clustering. Second, we compare our algorithm with modified SC with Vector Quantization (VQ), Hierarchical Clustering (HC)⁴ with Pivotal Sampling and HC with VQ for a sample size of 500. Since the results for modified SC with VQ come out to be closest to our algorithm, next, for broader appeal we compare these two algorithms for a sample size of 300. Third, we compare our algorithm with the current best in literature for this kind of data (i.e. HC without sampling) for both the sample sizes of 500 and 300. Fourth and finally, as discussed in the Literature Review section, we compare our sampling with that in Gireesh et al. (2015) and our clustering with the one in Islam et al. (2020).

Initially, we calculate the loss of accuracy incurred because of Pivotal Sampling in our algorithm. This loss for both the sample sizes and cluster size ten is listed in Table 7. Columns 1 and 2 give the sample sizes and the similarity measures chosen, respectively. Columns 3 and 4 give the Silhouette Values for modified SC without sampling (from Table 6) and our algorithm, respectively. The last column gives the percentage loss of accuracy. We can observe from this data that the loss of accuracy for one type of similarity measure (Correlation) is almost as low as −2% for both the sample sizes. This is considered acceptable because we are still better than the existing best algorithm (HC without sampling; please see Table 10 and its accompanying discussion below).

Here, we also perform a statistical test to support the above conjecture that using Pivotal Sampling does not substantially deteriorate the accuracy of our modified SC. For this, we use the ANOVA (analysis of variance) test (Rutherford, 2011). This test uses the variance between the different groups and the variance within each group to compute a value called the F-value, which is then compared with a standard estimate called F-critical. If F-value is less than F-critical, then it is inferred that the means of all the groups are equal.

The two groups for us refer to the modified SC results (column 3) and the modified SC with Pivotal Sampling results (column 4). The F-values here (using the Silhouette Values of the two groups) come out to be 0.3432 and 0.4202 for N = 500 and N = 300, respectively. Both these values are less than the F-critical value given in the F-distribution table of Beyer (2019), which is 5.9873. Thus, using the above mentioned ANOVA test theory, we infer that that the mean Silhouette Value of modified SC is similar to the mean Silhouette Value of modified SC with Pivotal Sampling for both the sample sizes.

The results for the second set of comparisons are given in Table 8. Columns 2 and 3 give the similarity measures and the number of clusters chosen, respectively. Columns 4 and 5 give Silhouette Values of modified SC with Pivotal Sampling and VQ, respectively, while columns 6 and 7 give Silhouette Values of HC with Pivotal Sampling and VQ, respectively.

When we compare our algorithm (values in the fourth column, and highlighted in bold) with other variants, it is evident that we are clearly better than modified SC with VQ and HC with VQ (values in the fifth and the seventh columns); all our values are higher than those from these two algorithms.

When we compare our algorithm with HC with Pivotal Sampling (values in the sixth column), we again perform better for many cases. However, for some cases, our algorithm performs worse than HC with Pivotal Sampling (highlighted with a *). Upon further analysis (discussed below), we realize that segregation of species by HC with Pivotal Sampling into fewer clusters than practically observed, results in these set of Silhouette Values getting wrongly inflated.

To further assess the quality of the proposed technique, we present the distribution of species into different clusters (after reverse-mapping) for HC with Pivotal Sampling and our algorithm. Without loss of generality, this comparison is done using the Squared Euclidean similarity measure and cluster size thirty. The results for HC with Pivotal Sampling are given in Fig. 2 and for our algorithm are given in Fig. 3. In both the figures, on the x-axis, we have the cluster number and on the y-axis, the number of species present in them.

As evident, Fig. 2 depicts a very skewed distribution, i.e. most species are segregated into only a few clusters, while the remaining clusters contain only one or two species. At a broader level, this biased distribution of species obtained by HC with Pivotal Sampling is correct since all species belong to the same plant. On the contrary, the distribution in Fig. 3 is fairly equal. That is, our algorithm equally distributes all species between the different clusters. At a finer level, this distribution is better since our algorithm is able to perform a more detailed clustering, i.e. it splits the bigger clusters into multiple smaller ones, which better captures the similarity between species.

This is also the reason for the inflation of Silhouette Values of HC with Pivotal Sampling in Table 8 since the intra-cluster similarity for solitary specie is zero leading to its respective Silhouette Value to become one (the maximum possible; see Eq. (21)). Thus, our algorithm also outperforms HC with Pivotal Sampling, which from Table 8 was not very evident.

Next, as mentioned earlier, to further demonstrate the applicability of our work, we also present the results with a sample size 300. Since modified SC with VQ turns out to be our closest competitor, we compare our algorithm with this one only. This comparison is given in Table 9, with its columns mapping the respective columns of Table 8. As evident from Table 9, our modified SC with Pivotal Sampling substantially outperforms modified SC with VQ (see values in columns 4 and 5).

As earlier, third, we compare the results of our algorithm (modified SC with Pivotal Sampling) with the currently popular clustering algorithm in the plant studies domain (i.e. HC without sampling). For this set of experiments, without loss of generality, we use the cluster size of ten. The results of this comparison are given in Table 10, where the first four columns are self-explanatory (based upon the data given in Tables 8 and 9 earlier). In the last column of this table, we also evaluate the percentage improvement in our algorithm over HC. As evident from this table, our algorithm is up to 45% more accurate than HC for both the sample sizes. As earlier, our algorithm also has the crucial added benefit of reduced computational complexity as compared to HC.

Fourth and finally, as mentioned in the Literature Review section, we also compare our work with two previous works that are closest to ours. With the dataset almost the same as used by us, that is, a slightly larger phenotypic data for Soybean species, Gireesh et al. (2015) performed Principal Component and Power Core based samplings to identify relationships between the different phenotypic characteristics (first category as in Section “Literature Review”). We compare our sampling results with the best from Gireesh et al. (2015) in Appendix B, which demonstrates the superiority of our sampling method. Islam et al. (2020) performed HC on phenotypic data for Rice species (second category as in Section “Literature Review”). In Appendix C, we apply modified SC on this dataset to again demonstrate that our clustering technique is better.

Conclusions and Future Work

We present the modified Spectral Clustering (SC) with Pivotal Sampling algorithm for clustering plant species using their phenotypic data. We use SC for its accurate clustering and Pivotal Sampling for its effective sample selection that in-turn makes our algorithm scalable for large data. Since building the similarity matrix is crucial for the SC algorithm, we exhaustively adapt seven similarity measures to build such a matrix. We also present a novel way of assigning probabilities to different species for Pivotal Sampling.

We perform four sets of experiments on about 2,400 Soybean species that demonstrate the superiority of our algorithm. First, we compare the Silhouette Values of modified SC without and with Pivotal Sampling, and show that the difference between these values is not significant. Second, when compared with the competitive clustering algorithms with samplings (SC with Vector Quantization (VQ), Hierarchical Clustering (HC) with Pivotal Sampling, and HC with VQ), Silhouette Values obtained when using our algorithm are higher. Third, our algorithm doubly outperforms the standard HC algorithm in terms of clustering accuracy and computational complexity. We are up to 45% more accurate and an order of magnitude faster than HC. Fourth and finally, we illustrate the excellence of our algorithm by comparing it with two previous works that are closest to ours.

Since the choice of the similarity matrix has a significant impact on the quality of clusters, in the future, we intend to adapt other ways of constructing this matrix such as Pearson χ², Squared χ², Bhattacharyya, Kullback-Liebler etc. (Cha, 2007). Furthermore, we also plan to observe the performance of Cube Sampling, which is another probabilistic sampling technique with data analysis properties complementary to Pivotal Sampling (Tille, 2006). Both Pivotal and Cube belong to the balanced sampling category, i.e. they satisfy Y ≈ Y′_HT and Y ≈ Y′_Hájek (recall Eqs. (18)–(20)). Cube Sampling automatically obtains the samples (without specifying the sample size), which does not happen in Pivotal. As mentioned earlier, our algorithm is developed to work well for phenotypic data of all plant species. This is because different species vary only in the number of characteristics and the type of characteristics, both of which do not affect our algorithm. We have preliminarily discussed this aspect for Maize and Rice in Appendix C, with extensive experiments for these two plants planned for future.

Appendix A

Here, we first present phenotypic data of the Soybean species used for our experiments. Please see Table A1 below. Next, we validate this data. For this, we compare our species data with a similar Soybean species data from (Gireesh et al., 2015) for the common set of phenotypic characteristics; Plant Height (PH), Number of Pods Per Plant (NPPP), and Days to Pod Initiation (DPI). This comparison is done using standard statistical metrics and is given in Table A2 below.

From this table, it is evident that the Standard Deviation (SD), Coefficient of Variance (CV), and Mean of our data and the data from the previous work are very close (for all three characteristics of PH, NPPP, and DPI). The slight variation in the metrics between the two data for all the characteristics is due to the difference in the ranges of the respective characteristics (due to the slightly differing selection of the species by the two works).

Appendix B

Here, we compare our sampling technique with those proposed by Gireesh et al. (2015) for a similar dataset. As earlier, we do Pivotal Sampling on 2,376 Soybean species while Gireesh et al. (2015) performed the Principal Component Score (PCS) and the Power Core (PC) samplings on 3,443 Soybean species. Since the samples obtained by the PC method are better, we compare our results with this sampling only.

This comparison is done using the statistical metrics of Standard Deviation (SD), Coefficient of Variance (CV) and Mean, and is given in Table A3 below. Since the metrics of our sampled data are more closer to our respective full data as compared to the metrics of the previous works’ sampled data to its respective full data, our sampling is better.

Appendix C

In the article, we have demonstrated the usefulness of our algorithm on the species of the Soybean plant, here we demonstrate our algorithms’ applicability to the species of the other two plants (Maize and Rice). The phenotypic data for the Maize species is given in Table A4, and for the Rice species is given in Table A5.

We can observe from Tables A1, A4, and A5 that there is a set of common phenotypic characteristics for the three plant species. Also, the values of all the characteristics are either categorical or numerical. As mentioned earlier, the categorical values can be easily converted to numerical ones. Since the input to our algorithm is a matrix built using the phenotypic data for given species, it can be applied to any of these plants.

To demonstrate the usefulness of our algorithm to the two new plant species, without loss of generality, we perform clustering of Rice species using our modified SC. For this, we use the data from Islam et al. (2020), where the authors have used HC to cluster ten Rice species into three clusters. Hence, we also cluster these ten species into three clusters using our modified SC. In Islam et al. (2020), the output is in the form of a hierarchical tree, which is non-numerical, and hence, difficult to compare. Thus, we compute Silhouette Values for our modified SC and HC. This data for the four similarity measures are given in Table A6. As evident from this table, our algorithm substantially outperforms HC.

Supplemental Information