Dynamic guided metric representation learning for multi-view clustering

Multi-view spectral clustering

Multi-view spectral clustering assumes that all the views share the same or similar eigenvector matrices, in which co-training spectral clustering and co-regularized spectral clustering are the two representative methods.

(1) Co-training spectral clustering. These algorithms are investigated under the assumption of consensus among multiple views and trained alternately to maximize the consistency of the two distinct views. Three main assumptions are made: (1) each view is sufficient for the learning task; (2) the views are conditionally independent given the class labels; and (3) the objective functions export the same predictions for co-occurring features with high probability in both views. Overall, most co-training methods are semi-supervised learning.

(2) Co-regularized spectral clustering. Zhu (e.g., Zhu, Zhang & He, 2019) proposed co-regularized approach. The main idea of this kind of method is to minimize the distinction between the predictor functions of two views acting as one part of an objective function. They used graphic Laplacian eigenvectors to play a role much like that of predictor functions in a semi-supervised learning scenario. Inspired by the previous work (e.g., Zhu, Zhang & He, 2019; Chen, Huang & Wang, 2020), automatically learned the weights of different views from data (e.g.,Ye, Liu & Yin, 2016) .

Multi-view subspace clustering

In practice, multi-view data can be sampled from multiple subspaces. A subspace learning model can learn a new and unified representation or a latent space for multi-view data. The unified representation or latent space can then be directly used for the clustering task. In addition, a subspace clustering model can deal with high-dimensional data. The approach is to find the underlying subspaces and then to cluster. Wang (e.g., Wang, Lin & Wu, 2015) proposed a model to measure correlation consensus in multiple views. Unlike Wang, Zhao (e.g., Zhao, Ding & Fu, 2017a; Zhao, Ding & Fu, 2017b) used a deep semi-nonnegative matrix factorization to perform clustering. This kind of method focus on mining the inherent structure from multiple subspace of views and the clustering performance is heavily dependent on the affinity matrix. Therefore, some works used deep networks to learn the inter-view specific features based on classical subspace clustering methods. The newest multi-view representation learning model (MRL) (e.g., Zheng, Ge & Li, 2020) is focus on mining the specific characteristics of inner-view and then learning the fusion representation based on the maximum correlation of multiple views. In all, how to mining the hidden difference of views is a research topic in recent years.

Multi-view non-negative matrix factorization clustering

Given a non-negative matrix X ∈ ℝ^d×n, non-negative matrix factorization (NMF) seeks two non-negative matrices W ∈ ℝ^d×p and V ∈ ℝ^p×n, whose product the original matrix X: ${\displaystyle \mathrm{X}\approx \mathrm{W}{\mathrm{V}}^{\mathrm{T}},}$

where W is the basis matrix and V the indicator matrix.

Due to its non-negativity constraints, NMF has emerged as a latent feature learning method. To combine multi-view information in an NMF framework, many variants of NMF have also been proposed. Kalayeh (e.g., Kalayeh, Idrees & Shah, 2014) present a weighted extension of Multi-view NMF to address the aforementioned drawbacks. Liu (e.g., Liu, Wang & Lu, 2020) used the cross entropy loss function to constrain the objective function better.

Multi-view multi-kernel clustering

Multi-kernel learning was originally used to combine views by means of a kernel and was widely used to deal with multi-view data. The common approach is to define a kernel for each view and then combine these kernels in a convex combination (e.g., Sellami & Alaya, 2021; Lu, Liu & Wei, 2020; Nithya, Appathurai & Venkatadri, 2020). Therefore, one main problem of this method is to choose kernel functions and optimal. A k-means analysis performed on kernel space that simultaneously found the best cluster labels, the cluster kernels, and the optimal combination of multiple kernels (e.g., Cui, Fern & Dy, 2007). Liu (e.g., Liu, Ji & Glänzel, 2013) extended k-means clustering into Hilbert space. Liu try to denote the data matrices as kernel matrices and then combined for fusion. In addition, consider the differences among views, methods with weighted combinations of kernels have also been studied.

However, multi-view data are incomplete. To address this issue, researchers integrated kernel imputation and clustering into a unified learning procedure (e.g., Monney, Zhan & Zhen, 2020). Besides, other methods use a direct combination of features to perform multi-view clustering, like those in (e.g., Xu, Han & Nie, 2016; Nie, Li & Li, 2017).

Multi-view CCA-based methods clustering

For multi-view data clustering, it is reasonable to combine the data directly. However, it is hard to fuse different data types, and high dimensionality and noise are difficult to handle. The CCA method is used to combine the views directly and CCA-based techniques for multi-view clustering fusion. Chaudhuri (e.g., Chaudhuri, Kakade & Livescu, 2009) projected the data into a lower-dimensional space using CCA and cluster. Then, he CCA-based methods are proposed, such as KCCA (e.g., Wang, Li & Huang, 2017), DCCA(e.g., Andrew, Arora & Bilmes, 2013), DGCCA (e.g., Benton, Khayrallah & Gujral, 2017) and MRL(e.g., Zheng, Ge & Li, 2020). Overall, although DCCA-based methods are used now in multi-view representation, this is a direction worthy of further multi-view clustering study using DCCA-based methods. In contrast to these methods, the representation will directly influence effectiveness on the clustering task. Thus, based on existing multi-view clustering learning methods, the intention is to design a novel model to learn a comprehensive meaningful data representation with the discriminant characteristics in a single view and correlation of all views.

In short, existing methods still has the following challenges: (1) Common matrix methods focus on learning a shared similar structure, they ignore the hidden difference of views. (2) View fusion methods considering the unique features of each view and the correlation of all views. However, existing methods strategy is two steps. The first step is self-representation of each view independently and the next step is focus on fusion views. The two-step collaboration is not well meet the clustering task.

Network architecture

Figure 1 illustrates the architecture of the proposed DGMRL-MVC model for multi-view clustering. This architecture effectively improves fusion representation and overcomes the limitations of traditional DCCA-based multi-view clustering learning. The model consists of three modules: inter-intra representation based on Fisher discriminant analysis and the Hilbert–Schmidt independence criteria, together called FDA-FISH metric learning; deep representation based on dynamic guided deep learning; shared representation based on deep generalized canonical correlation analysis. The first module learns intra-view separability and inter-view consistency. Then, as for dynamic guided deep learning, it is used to introduces location or direction information to enhance the single view representation. In the last sub-model, the fusion representation G of multiple views for the clustering task is created.

1) Inter-intra representation based on FDA-HSIC

The Fisher-HSIC Multi-View Metric Learning (FISH-MML) system is proposed in the work (e.g., Zhang, Liu & Liu, 2018). This method is based on FDA and HSIC, which are simple, yet rather effective. Inspired by Zhang’s work, an effort was made to add FDA-HSIC into the proposed framework for multi-view clustering representation learning.

Intra-view separability. The starting point is the definitions of between-class and total scatter matrices in the v^th view: (2)${\displaystyle S_b^{\left(v\right)}=\frac{1}{n}\sum _{j=1}^m{n}_j\left({\mu }_j^{\left(v\right)}-{\mu }^{\left(v\right)}\right){\left({\mu }_j^{\left(v\right)}-{\mu }^{\left(v\right)}\right)}^T,}$ (3)${\displaystyle S_t^{\left(v\right)}=\frac{1}{n}\sum _{i=1}^n\left(z_i^{\left(v\right)}-{\mu }^{\left(v\right)}\right){\left(z_i^{\left(v\right)}-{\mu }^{\left(v\right)}\right)}^T,}$ (4)${\displaystyle {\mu }_j^{\left(v\right)}=\frac{1}{n_j}\sum _{i=1}^{n_j}{z}_i^{j\left(v\right)},{\mu }^{\left(v\right)}=\frac{1}{n}\sum _{i=1}^{n_j}{z}_i^{\left(v\right)},}$ where b is the abbreviation of between, t is the abbreviation of total, m is the number of classes, n_j is the number of samples belonging to class C_j, ${\mu }_j^{\left(v\right)}$ are the sample means of the v^th view for class C_j, and ${\mu }^{\left(v\right)}$ are the sample means of the v^th view for all the views. $z_{{i^j}^{\left(v\right)}}$ is the projected feature vector corresponding to $x_{{i^j}^{\left(v\right)}}$ andis defined as follows: (5)${\displaystyle z_{{i^j}^{\left(v\right)}}=P^{\left(v\right)}{x}_i^{j\left(v\right)},}$

where the symmetric positive semi-definite matrix M = P^TP andP ∈ ℝ^k×d(k ≥ rank(M) is the new space.

The trace operator for ${\mathrm{S}}_{\mathrm{b}}^{\left(\mathrm{v}\right)}$ and ${\mathrm{S}}_{\mathrm{t}}^{\left(\mathrm{v}\right)}$conditioned on ${\mathbf{P}}^{\left(\mathrm{v}\right)}$ is then optimized by the following optimization function: (6)${\displaystyle \underset{{\left\{P^{\left(v\right)}\right\}}_{v=1}^V}{max}\sum _{v=1}^{V}Tr\left({\mathbf{S}}_b^{\left(v\right)};{\mathbf{P}}^{\left(v\right)}\right)-\gamma Tr\left({\mathbf{S}}_t^{\left(v\right)};{\mathbf{P}}^{\left(\mathrm{v}\right)}\right),}$

where γ is a tunable parameter to balance the two terms involved. Therefore, the above optimization function is focused on seeking metrics that jointly maximize separability.

Inter-view consistency. The next step is to explore the complementarity information from multi-views using HSIC (e.g., Gretton, Bousquet & Smola, 2005). Based on HSIC, the function can be defined as: (7)${\displaystyle \text{HSCI}\left({\mathbf{Z}}^{\left(v\right)},{\mathbf{Z}}^{\left(w\right)}\right)={\left(n-1\right)}^{-2}\mathrm{tr}\left({\mathbf{K}}_v\mathbf{H}{\mathbf{K}}_w\mathbf{H}\right),}$ (8)${\displaystyle {\mathbf{K}}^{\left(v\right)}={{\mathbf{Z}}^{\left(v\right)}}^T{\mathbf{Z}}^{\left(v\right)}={{\mathbf{X}}^{\left(v\right)}}^T{{\mathbf{P}}^{\left(v\right)}}^T{\mathbf{P}}^{\left(v\right)}{\mathbf{X}}^{\left(v\right)},}$ (9)${\displaystyle {\mathbf{K}}^{\left(w\right)}={{\mathbf{Z}}^{\left(w\right)}}^T{\mathbf{Z}}^{\left(w\right)}={{\mathbf{X}}^{\left(w\right)}}^T{{\mathbf{P}}^{\left(w\right)}}^T{\mathbf{P}}^{\left(w\right)}{\mathbf{X}}^{\left(w\right)},}$ where ${\mathbf{K}}^{\left(v\right)}$ and ${\mathbf{K}}^{\left(w\right)}$ are the Gram matrices from the two views parameterized by the projections ${\mathbf{P}}^{\left(v\right)}$ and ${\mathbf{P}}^{\left(w\right)}$. In the model, the dependency between ${\mathbf{K}}^{\left(v\right)}$ and ${\mathbf{K}}^{\left(w\right)}$ is enhanced by maximizing the HSIC function, and H is the Gram matrix that ensures zero mean in the feature space.

Optimization. The FDA-HSIC optimization function can be described as: ${\displaystyle \underset{{\left\{P^{\left(v\right)}\right\}}_{v=1}^V}{max}\sum _{v=1}^{V}Tr\left({\mathbf{S}}_{\mathbf{b}}^{\left(\mathbf{v}\right)};P^{\left(v\right)}\right)+{\lambda }_{1}Tr\left({\mathbf{S}}_{\mathbf{t}}^{\left(\mathbf{v}\right)};P^{\left(v\right)}\right)+{\lambda }_2\sum _{v\ne w}\text{HSIC}\left({\mathbf{P}}^{\left(v\right)}{\mathbf{X}}^{\left(v\right)},{\mathbf{P}}^{\left(w\right)}{\mathbf{X}}^{\left(w\right)}\right)}$

= ${max}_{{\left\{P^{\left(v\right)}\right\}}_{v=1}^V}{\sum }_{v=1}^{V}Tr\left(P^{\left(v\right)}\left(\mathbf{A}+{\lambda }_1\mathbf{B}+{\lambda }_2\mathbf{C}\right){{\mathbf{P}}^{\left(v\right)}}^{\mathrm{T}}\right)={max}_{{\left\{P^{\left(v\right)}\right\}}_{v=1}^V}{\sum }_{v=1}^{V}tr\left({\mathbf{P}}^{\left(v\right)}\mathbf{D}{{\mathbf{P}}^{\left(v\right)}}^T\right)$ (10)${\displaystyle s.t.{\mathbf{P}}^{\left(v\right)}{{\mathbf{P}}^{\left(v\right)}}^{\mathrm{T}}=\mathbf{I},\mathrm{v}=1,2,\dots ,\mathrm{V},}$

where λ₁ > 0 and λ₂ > 0.

2) Deep representation based on dynamic guided learning

The output ${\mathbf{P}}^{\left(v\right)}$of first submodel is fed into this submodel. ${\mathbf{P}}^{\left(v\right)}$ is the v^th view. In this process, the learning of each view is independent. Inspired by Sabour’s work (e.g., Sabour, Frosst & Hinton, 2017), this submode adopt the dynamic guided mechanism to learn the location or direction information of view. Taking the first view ${\mathbf{P}}^{\left(1\right)}$ as example, we first divide ${\mathbf{P}}^{\left(1\right)}$ into several capsules u₁, …, u_k1. Each capsule u_k1 by a weight matrix ${\mathrm{W}}_{{\mathrm{k}}_1{\mathrm{k}}_2}^{\mathrm{\ast }}$ and get a prediction vector ${\hat{\mathrm{o}}}_{{\mathrm{k}}_2\left|{\mathrm{k}}_1\right.}.{\hat{\mathrm{o}}}_{{\mathrm{k}}_2\left|{\mathrm{k}}_1\right.}$ can be called deep learning mapping. Next, to represent the probability for the output vector, a nonlinear function squashing is applied to s_k2.And the new capsule s_k2 is weighted sum over u_k1 with the coupling coefficient c_k1k2. (11)${\displaystyle {\hat{\mathrm{o}}}_{{\mathrm{k}}_2\left|{\mathrm{k}}_1\right.}={\mathrm{W}}_{{\mathrm{k}}_1{\mathrm{k}}_2}^{\mathrm{\ast }}{\mathrm{u}}_{{\mathrm{k}}_1},}$ (12)${\displaystyle {\mathrm{s}}_{{\mathrm{k}}_2}=\sum _{\mathrm{k}1}{\mathrm{c}}_{{\mathrm{k}}_1{\mathrm{k}}_2}{\hat{\mathrm{o}}}_{{\mathrm{k}}_2\left|{\mathrm{k}}_1\right.},}$ (13)${\displaystyle {\mathrm{c}}_{{\mathrm{k}}_1{\mathrm{k}}_2}=\text{softmax}\left(p_{k_1{k}_2}\right)=\frac{exp\left(p_{k_1{k}_2}^{\ast }\right)}{\sum _{\mathrm{k}}exp\left(p_{k_{1}k}^{\ast }\right)},}$ where the initial logits $p_{k_1{k}_2}^{\ast }$ is the log prior probabilities that capsule k₂ to k₁.

Finally, the output of the first view is P^∗(1), it is denoted as: P^∗(1) = [v₁, v₂, ……, v_k2], where v_k2 = [v₁, …, v_j]. The discriminant characteristics of single view is learned: (14)${\displaystyle {\mathrm{v}}_{{\mathbf{k}}_2}=\text{squas}h\left({\mathrm{s}}_{{\mathrm{k}}_2}\right)=\frac{{∥{\mathrm{s}}_{{\mathrm{k}}_2}∥}^2}{1+{∥{\mathrm{s}}_{{\mathrm{k}}_2}∥}^2}\frac{{\mathrm{s}}_{{\mathrm{k}}_2}}{∥{\mathrm{s}}_{{\mathrm{k}}_2}∥},}$ 3) Shared representation based on deep generalized canonical correlation analysis

This sub-model learns the ultimate common fusion representation G from all views. In this process, minimized loss function as follows: ${\displaystyle \underset{{\mathrm{U}}_v\in {\mathbb{R}}^{{\mathrm{d}}_v\times \mathrm{r}},G\in {\mathbb{R}}^{\mathrm{r}\times \mathrm{N}}}{\text{minimize}}\sum _{v=1}^{\mathrm{V}}{∥\mathbf{G}-{\mathbf{U}}_{\mathbf{v}}^{\mathbf{T}}{\mathbf{P}}^{\ast \left(v\right)}∥}_{\mathrm{F}}^2,}$ (15)${\displaystyle subjectto\mathbf{G}{\mathbf{G}}^{\mathbf{T}}={\mathbf{I}}_{\mathbf{r}},}$ where P^∗(v) ∈ ℝ^d_v×N_v is latent discriminated embedding of the v^th view. Then, all ${\mathrm{P}}_v^{\mathrm{\ast }}$ spliced into a matrix G.

Training

In this model, the scaled empirical co-variance matrix C_vv of the v^th network is calculated by C_vv = P^∗(v)(P^∗(v))^T ∈ ℝ^d_v∗d_v. Where ${\mathbf{P}}^{\ast \left(v\right)}={{\mathbf{P}}^{\ast \left(v\right)}}^{\mathbf{T}}{\mathbf{C}}_{vv}^{-1}{\mathbf{P}}^{\ast \left(v\right)}\in {\mathbb{R}}^{\mathrm{N\ast N}}$ is the corresponding projection matrix. In optimization process, we consider maximize the sum of correlations between fusion representation G and single view. The reconstruction error as follows: ${\displaystyle \sum _{v=1}^{\mathrm{V}}{∥\mathbf{G}-{\mathbf{U}}_v^{\mathbf{T}}{\mathbf{P}}^{\ast \left(v\right)}∥}_{\mathrm{F}}^2=\sum _{v=1}^{\mathrm{V}}{∥\mathbf{G}-\mathbf{G}{{\mathbf{P}}^{\ast \left(v\right)}}^{\mathbf{T}}{\mathbf{C}}_{vv}^{-1}{\mathbf{P}}^{\ast \left(v\right)}∥}_{\mathrm{F}}^2}$ (16)${\displaystyle =r\mathbf{J}-\mathrm{Tr}\left(\mathbf{GM}{\mathbf{G}}^T\right),}$ where the positive semi-definite matrix of each view is $\mathbf{M}={\sum }_{v=1}^{\mathbf{V}}{\mathbf{P}}_v$ , r is the top rows of M. To learn the correlation of every pair of views, we stack them in a j∗j matrix, it is defined as J.

The derivative of loss function is: (17)${\displaystyle \frac{\partial \mathbf{L}}{\partial {\mathbf{P}}_v^{\ast }}=2{\mathbf{U}}_{v}G-2{\mathbf{U}}_v{\mathbf{U}}_v^{\mathbf{T}}{\mathbf{O}}_v^{\ast },}$

where $\mathrm{L}={\sum }_{\mathrm{i}=1}^{\mathrm{r}}{\lambda }_{\mathrm{i}}\left(\mathbf{M}\right)$.

(RQ1) How good is the performance improvement of the proposed model with dynamic guided representation? (DGMRL-MVC vs. FISH-MML)

FISH-MML provided the best multi-view metric learning with FDA-FISH. The proposed DGMRL-MVC model was first compared with FISH-MML. The results showed that DGMRL-MVC significantly outperformed FISH-MML on all four datasets for the multi-view clustering task. Table 2 presents the comprehensive comparison results. It is obvious that the proposed model achieved the best performance on nearly all datasets under all metrics. Take the Handwritten dataset with six views as an example, the improvement of DGMRL-MVC over FISH-MML was about 42.2%, 60.4%, 46%, 60.3% and 19.5% in terms of precision, recall, F1, RI, and NMI respectively. This proved that the dynamic guided representation used in the proposed method is effective for multi-view clustering. Our model has been enhanced in two aspects, i.e., inter-intra relations and dynamic routing learning embedded into each view self-representation, and fusion representation with maximum correlation.

(RQ2) How good is the performance of DGMRL-MVC compared with other DCCA-based methods?

The latent DCCA-based representation was compared with the proposed algorithm on a clustering task. DCCA and DGCCA were chosen as baselines. In the dynamic guided representation-MVC model, FDS-FISH was not present, and only dynamic guided representation and generalized canonical correlation analysis were added. As shown in Table 3, the results showed that DGMRL-MVC significantly outperformed DGCCA and DCCA on all datasets. In addition, the performance using the proposed discriminated representation was generally better than that using dynamic guided representation-MVC. In all, our model achieved comprehensive results because our model adds FDA-FISH and dynamic guided module that distills more separable intra-view and specific intrinsic features, resulting in a high-quality latent discriminated embedding.

(RQ3) Is the discriminated fusion representation good on the clustering task?

As shown in Tables 4 and 5, the visualization is consistent with the clustering results. In this experiment, the Silhouette_score was also chosen as the evaluation indicator. Tables 4 and 5 reveal the advantage of the proposed model when the dynamic guided and FDA-HSIC models were used simultaneously. The Silhouette_score for clustering was higher than that for the FISH-MML model. Besides, the Silhouette_score of the full view method was higher than that of the 2-view method, which proved the necessity of the multi-view fusion representation. Our model has better learning performance by using full views, and can effectively mine the features of multiple views. The result empirically proves that clustering with full-view is more robust that with 2-view.

(RQ4) How does performance vary w.r.t. the parameter of n_clusters?

Parameter analysis was conducted on the Silhouette_score parameter by varying the number of clusters in [2,3,5,7,8,9,10] (Handwritten and Wikipedia datasets) and [2,5,8,10,13,15,18,20] (Football and Pascal datasets). Figures 2, 3, 4 and 5 plots the results in terms of Silhouette_score when different numbers of clusters were used on the four datasets. The proposed DGMRL-MVC showed the best performance on the other datasets. However, DGMRL-MVC slightly worse than the other methods on the Handwritten dataset. Further, combined with the previous experimental results, we analysis the reason of Fig. 2 result. First, this dataset is different from other datasets, its views are six types of features on same picture, i.e., Fourier coefficients of the character shapes, profile correlations, Karhunen-love coefficients, pixel averages in 2 × 3 windows, Zernike moment, morphological. Due to the large difference and weak correlation between the original data views, our model weakens the original relationship of views. The value of the Silhouette_score reflects the distance of samples in same cluster. In detail, although clustering evaluation metrics of our model is higher than the baselines, the sample is closer to the boundary of the cluster in this kind of dataset.

Table 4:

Visualization of clustering result with FISH-MML on the Pascal dataset.

The best results are highlighted in bold.

DOI: 10.7717/peerjcs.922/table-4

Table 5:

Visualization of clustering result with DGMRL-MVC (ours) on the Pascal dataset.

The best results are highlighted in bold.

DOI: 10.7717/peerjcs.922/table-5

(RQ5) Ablation experiment

Finally, ablation experiments were used to further verify the effectiveness of the model. We compared the differences of model learning performance under three steps respectively. Specifically, Step-A is inter-intra representation, Step-B is fusion representation directly, Step-C is dynamic routing and fusion representation. From the results of Fig. 6, we can find that performance of Step-B is lower than Step-C. The main reason is that fusion learning is only aimed at the maximum correlation between views, lacking difference learning of views, which is not conducive to the feature expression for clustering tasks. At the same time, the performance difference between Step-B and Step-A is not big, and only some indicators are slightly higher than Step-A, indicating that in the process of fusion learning, the specific intrinsic features learning with dynamice routing plays the most obvious role. Our method result is higher than other sub-models, which proves that on the basic of strengthening the learning view relationship, adding subspace fusion characterization can effectively improve the clustering performance of model.