On component-wise dissimilarity measures and metric properties in pattern recognition

Introduction

In the past few decades, the discipline of pattern recognition (PR), aiming to automatically discover regularities in data, focused most efforts in frameworks conceived to learn from examples, thus from observations. These frameworks exploit several machine learning techniques grounding on the data-driven approach (Bishop, 2006). Therefore, in these specific cases, the goal of a PR system is to find regularities in data aiming to reach good generalization capabilities by building a model from known observations (Hart, Stork & Duda, 2000). Thereby, at the basis of an automated PR pipeline there are the observations, that can be any type of measurements on real-world objects. Observations can be collected by hand or automatically by sensors. Furthermore, observations can be labeled and labels allow to distinguish the class or the category in which the object falls (Martino, Giuliani & Rizzi, 2018). Moving away from the “philosophical” problem among the differences between objects that live in the real world—a discussion that deserves a systematic and really interesting discussion—it can be stated that it is really difficult to enumerate all differences between two real-world objects, at least, at raw (atomic) level. So what we can reveal are the differences between the (physical or virtual) properties of two objects and tell whether they can be considered different. This task takes part to the PR process and deserves a thorough discussion. In the PR jargon, the problem is known as finding a good representation of objects, e.g., if the weight is important as a property defining the objects, it should be taken into account, otherwise the system should not consider the “weight” feature. A really interesting theoretical treatment, within the context of cognitive science, on how natural properties arise from generic objects in building a suitable representation is provided by Peter Gärdenfors with the theory of conceptual spaces (Gärdenfors, 2004).

A representation can exist in several forms, such as numbers, strings, graphs, images, spectra, time series, densities and similarities (Duin & Pękalska, 2012). Robert P. W. Duin states that (Pękalska & Duin, 2012): (i) “every real-world difference between objects that may play a role in the human judgment of their similarity should make a difference in the representation” and ii) “the representation of a real world object, i.e., the mapping from the object to its representation, should be continuous”. Hence, these prescriptions indicate that the representation should consider real-world properties judged as important and, furthermore, two similar objects should be similar in their representations too. On the top of a good representation, it is possible to train a myriad of learning algorithms capable to generate a model from data objects and, finally, to generalize towards previously unseen data. In fully supervised learning, the generalization process (classification) needs labeled examples, while unlabeled examples are used in unsupervised learning schemes (Jain, Murty & Flynn, 1999; Jain, Duin & Mao, 2000). As we will see, alongside the classical learning algorithms adopted in machine learning, it can be useful to learn a dissimilarity function tailored to the data at hand. This particular task belongs to the metric learning (ML) paradigm, a florid research field in PR (Lu et al., 2018; Bengio, Courville & Vincent, 2013).

As anticipated, many real-world objects in PR cannot be simply described by a set of measurements collected in real-valued vectors. In other words, the representation of objects may not easily start from a vectorial space and in this case the dissimilarity measure cannot be simply defined as a plain Minkowski distance, for example. In this case, a data structure, known as dissimilarity matrix, becomes clearly important. Thereby, in many cases, the core of a PR system is a custom-based dissimilarity measure, that is a way to measure the dissimilarity between samples of a given complex process that are described by a set of measurements that can (even simultaneously) involve real numbers, integers, vectors, categorical variables, graphs, spectra, histograms, unevenly objects/events sequences, time series etc. This happens when real-world objects possess a complex description arising from different intrinsic characteristics, each one caught by a suitable data structure. Thence, the overall dissimilarity can be chosen within the family of Euclidean distances, or within the general class of Minkowski distances. However, the structure of the given distance needs to take into account the different data structures. In technical literature, distances involving complex and possibly heterogeneous data structures are known as component-wise or element-wise distances (Jimenez, Gonzalez & Gelbukh, 2016) where, for each component, it is used a specific difference operator for the data structure at hand and, once collected, all of them are synthesized in a “template” distance that may have the Euclidean or Minkowski general form. In other words, distances are a function of the additive combination of the contributions of their components (Beals, Krantz & Tversky, 1968). A further generalization can be derived from the weighted Euclidean distance (WED) where a weight is associated to each component. In ML tasks, these weights can be suitably learned automatically, usually through an optimization procedure.

The WED is widely applied in PR problems such as in bioinformatics and personalized medicine (Hu & Yan, 2007; Martino et al., 2020; Di Noia et al., 2020), speech synthesis (Lei, Ling & Dai, 2010) or in the industrial field (Rao, 2012). For example, WED is used in clustering application dealing with side information (Xing et al., 2002). In fact, if a clustering algorithm, such as k-means, initially fails to find a meaningful solution for the problem at hand from the user point of view, the user is forced to manually tweak the metric until sufficiently good clusters are found. In Schultz & Joachims (2003), the authors present a method for learning a distance metric starting from relative comparison such as “A is closer to B than A is to C”. A similar application can be found in (Kumar & Kummamuru, 2008), where a local metric is learned.

Moreover, in many real-world problems dealing with complex systems, the starting space is not a vectorial space, being also often non-metric (e.g., in life sciences (Münch et al., 2020; Martino, Giuliani & Rizzi, 2018), engineering applications (D’urso & Massari, 2019; De Santis, Arnò & Rizzi, 2022; Kim, Lee & Kim, 2018) or cybersecurity (Granato et al., 2020, 2022)). Consequently, only the dissimilarity representation is available through the dissimilarity matrix, as stated above. Hence, in such cases, the dissimilarity matrix is a primitive data structure compared to the data matrix. As we will see in the following, a dissimilarity matrix $\mathcal{D}$ is said to be “Euclidean” if it is perfectly (isometrically) embeddable in an Euclidean vector space in which the distances calculated in the latter are identical to the ones belonging to the entries of $\mathcal{D}$ (De Santis, Rizzi & Sadeghian, 2018). Several standard classifiers are designed to work effectively on Euclidean vector spaces. Operating with a non-Euclidean (or even non-metric) dissimilarity matrix may cause some problems. As an example, a non-Euclidean distance matrix leads to a non-positive definite kernel and the quadratic optimization procedure used to train a support vector machine (Vapnik, 1998; Schölkopf, Burges & Smola, 1999) may thereby fail, not being fulfilled the Mercer conditions (Mercer, 1909; Duin, Pękalska & Loog, 2013; Pękalska & Duin, 2005). However, in order to train standard classifiers on this kind of data, some solutions can be found. The two main solutions are based either on considering the dissimilarity matrix as the starting vector space endowed with the standard Euclidean distance (dissimilarity space representation) or by adopting a suitable transformation of the dissimilarity matrix, leading to the Pseudo-Euclidean (PE) space (Pękalska & Duin, 2005; De Santis et al., 2018; De Santis, Rizzi & Sadeghian, 2018). In the current study, we will consider the last case. For the first case, the interested reader can be referred to Pękalska & Duin (2005).

It is well known that from dissimilarity data collected in form of a dissimilarity matrix $\mathcal{D}$ it can be “reconstructed” the starting Euclidean space where the original data points lie (De Santis, Rizzi & Sadeghian, 2018). The reconstruction process (known as embedding) tries to generate the original vector space such that the distances are preserved as well as possible. Classical multi-dimensional scaling is an example of such embedding procedure (Borg & Groenen, 2005). For an Euclidean space all the distances are preserved and thus an Euclidean distance matrix can be embedded isometrically in an Euclidean space. For non-Euclidean distance matrices the Euclidean space is not “large enough” to embed the dissimilarity data even if they can be still embedded in the so called PE space (Goldfarb, 1984). The embedding procedure involves the eigendecomposition of the kernel matrix $\mathbf{G}=\mathbf{X}{\mathbf{X}}^T$, where $\mathbf{X}$ is the configuration matrix with data points organized as rows, also known as the Gram matrix (Horn & Johnson, 2013). The latter is a similarity matrix, obtainable through a suitable linear transformation of the dissimilarity matrix $\mathcal{D}$.

In this work, we consider a class of PR problems involving a dissimilarity matrix $\mathcal{D}$ deriving from a custom-based component-wise dissimilarity measure $d(x,y;\mathbf{w}):\mathcal{F}\times \mathcal{F}\to {\mathbb{R}}^{+}$. The following study is based on the characterization of d as a composite dissimilarity matrix of the form: $\sqrt{{\overline{\mathbf{d}}}_c^T{\mathbf{W}}^T\mathbf{W}{\overline{\mathbf{d}}}_c}$ computed as the $\mathbf{\ell }$₂ norm of the vector ${\overline{\mathbf{d}}}_c$ that collects the component wise (sub)-dissimilarity measures, of which the functional form is related to the specific features (i.e., data structure) within a suitable structured non-metric feature space $\mathcal{F}$.

Within this framework, in this article we provide two characterizations. The first one tries facing the claim according to which the behavior of a general dissimilarity measure depends on the behavior of the component-wise (sub)-dissimilarities. Specifically, d generates an Euclidean dissimilarity matrix if the (sub)-dissimilarities $d_{{\mathcal{F}}_j}$ are Euclidean. Therefore, the features ${\mathcal{F}}_j$ over which it is induced a particular dissimilarity measure, i.e., a structural dissimilarity in the sense of Duin & Pękalska (2010), can influence the nature of the mathematical space where the learning algorithm works.

As concerns the second characterization, it is really interesting to arrange a mathematical interpretation of the weights pertaining the custom-based dissimilarity matrix, in particular wondering what is the influence of a weighting matrix $\mathbf{W}$ on the eigenspectrum of the underlying Gram matrix ${\mathbf{G}}_w$, that is the Gram matrix obtained from the weighted version of the dissimilarity matrix $\mathcal{D}$. Unfortunately the relationship between the eigenvalues of $\mathbf{G}$ and ${\mathbf{G}}_w$ in a general case is not straightforward, being an open problem of mathematics (Zhang & Zhang, 2006; Fulton, 2000). It is approachable in particular cases of commuting matrices (in the case of matrices sharing a complete set of eigenvectors, i.e., normal matrices) or when one of the two is a scalar matrix, i.e., a matrix of the form $\mathbf{W}=k\mathbf{I}$. We will trace some results in the latter case.

Although this article aims at addressing these characterizations via a theoretical and mathematical viewpoint, the interested reader can find practical applications in the following articles. In De Santis et al. (2015), De Santis, Rizzi & Sadeghian (2018) a One-Class classification approach is used in the field of predictive maintenance and in the real-time recognition of faults in a real-world power grid, by processing heterogeneous information coming from smart sensors related to the power grid equipment and to the surrounding environment. The system exploits a clustering-genetic algorithm (GA) (Goldberg, 1989) approach where the weights of a custom based Euclidean dissimilarity measure are learned solving a suitable optimization problem. In De Santis et al. (2018), we addressed the problem of finding suitable representative elements in the dissimilarity space¹ in order to classify protein contact networks according to their enzymatic properties and in De Santis et al. (2022), the dissimilarity space embedding has been used to recognize signals pertaining to malfunctioning states of pressurization systems for high-speed railway trains. Finally, in Martino et al. (2020) the same problem of classifying protein contact networks according to their enzymatic properties has been solved by an hybridization of dissimilarity spaces and multiple kernel learning.

The degree of “non-metricity” and even of non-Euclidean behavior can be measured suitably with specific indexes obtained from the PE embedding such as the Eigen-Ratio, the Negative Eigen-Fraction and the Non-Metricity Fraction, each of which measures the non-Euclidean behavior, e.g., of a given dissimilarity matrix (Pękalska et al., 2006). Therefore, while the second question concerns the relation between $\mathbf{G}$ and ${\mathbf{G}}_w$, in the first characterization we are wondering what is the influence of dissimilarity weights on the Negative Eigen-Fraction, hence we are questioning on how it is possible to tune the non-Euclidean behavior of a custom-based dissimilarity matrix.

The article is organized as follows. In “Metric Learning” it is provided a brief review of the various ML paradigms treated in the literature. “On Metric Spaces and Dissimilarity Matrices” is a concise description of metric spaces and related dissimilarity matrices that serves as background. “The Weighted Euclidean Distance” is a deepening of the Euclidean distance structure and its weighted component-wise counterpart. “Characterization of a Composite Component-wise Dissimilarity” and “On the Presence of Weights in a Component-wise Dissimilarity and the Eigenspectrum of the Gram Matrix” sketch an experimental evaluation of the proposed principal investigations and, finally, “Conclusion” concludes the article.

Metric learning

The ML problem is concerned with learning a distance function tuned to a particular task and has been shown to be useful when exploited in conjunction with techniques relying explicitly on distances or dissimilarities, such as clustering algorithms, nearest-neighbor classifiers, etc. For example, if the task is to asses the similarity (or dissimilarity) between two images with the aim of finding a match, e.g., in face recognition, we would discover a proper distance function that emphasizes appropriate features (hair color, ratios of distances between facial key-points, etc.). Although this task can be performed by hand, it is very useful to develop tools for learning automatically the subset of meaningful features for the problem at hand. In fact, as anticipated in “Introduction”, useful representations can be also learned. However, it is unquestionable that, at least on a theoretical level, representation learning must be taken separate from classification tasks as depicted in Fig. 1 and discussed in Bellet, Habrard & Sebban (2013).

The ML step can be conceived as a first step in the open-loop pipeline depicted in Fig. 1, to be performed before the model synthesis stage. Moreover, both tasks can be done together in the same system, representing an advanced closed-loop and automatic PR system. It is the case, for example, of feature selection and feature extraction techniques. The last procedures can be done manually, but if they are automated (i.e., optimized) they can be a building block of the classification system itself. There are many methodologies capable to learn a representation; some authors distinguish between neural learning, that is learning by means of deep learning techniques, and ML. Despite this distinction, in general, both approaches ground the learning procedure on optimization techniques. Neural learning is useful in finding a good feature space, while ML involves the learning of suitable manifold where data objects lie and where they can be well represented for solving the problem at hand.

Many declinations of ML are available and, according to Fig. 2, they can be resumed in three principal paradigms: fully supervised, weakly supervised and semi supervised. An informal formulation of the supervised ML task is as follows: given an input distance function $d(\mathbf{x},\mathbf{y})$ between objects $\mathbf{x}$ and $\mathbf{y}$ (for example, the Euclidean distance), along with supervised information regarding an ideal distance, construct a new distance function $\hat{d}(\mathbf{x},\mathbf{y})$ which is “better” than the original distance function (Kulis, 2012). Normally, fully supervised paradigms have access to a set of labeled training instances, whose labels are used to generate a set of constraints. In other words supervised ML is cast into pairwise constraints: the equivalence constraints where pairs of data points belong to the same classes, and inequivalence constraints where pairs of data points belong to different classes (Bar-Hillel et al., 2003; Xing et al., 2002). In weakly supervised learning algorithms we do not have to access to the label of individual training examples and learning constraints are given in a different form as side information, while semi-supervised paradigms do not use either labeled samples or side information. Some authors (e.g., Saul & Roweis (2003)) deal with unsupervised ML paradigms, sometimes called also manifold learning, referring to the idea of learning an underlying low-dimensional manifold² where geometric relationships (e.g., the distance) between most of the observed data are preserved. Often this paradigm coincides with the dimensionality reduction paradigm such as the well-known Principal Component Analysis (PCA) (Shlens, 2014; Giuliani, 2017) and the Classical Multi-Dimensional Scaling, based on linear relations. As concerns non-linear counterparts, it is worth taking note of embedding methods such as ISOMAP (Tenenbaum, De Silva & Langford, 2000), Locally Linear Embedding (Roweis & Saul, 2000) and Laplacian Eigenmap (Belkin & Niyogi, 2003). Other methods are based on information-theoretic relations such as the Mutual Information. Hence, the form or structure of the learned metric can be linear, non-linear, local. Linear ML paradigms are based on the learning of a metric in the form of a generalized Mahalanobis distance (Mahalanobis, 1936) between data objects, i.e., ${\mathcal{D}}_{ij}^{\mathbf{W}}=\sqrt{{\left({\mathbf{x}}_i-{\mathbf{x}}_j\right)}^T{\mathbf{W}}^T\mathbf{W}({\mathbf{x}}_i-{\mathbf{x}}_j)}=\sqrt{\parallel \mathbf{W}({\mathbf{x}}_i-{\mathbf{x}}_j){\parallel }_2}$, where $\mathbf{M}={\mathbf{W}}^T\mathbf{W}$ is a matrix with suitable properties that has to be learned. In other words, the learning algorithm learns a linear transformation $\mathbf{x}\to \mathbf{W}\mathbf{x}$ that better represents the similarity in the target domain. Sometimes, there are some non-linear structures in the available data that linear algorithms are unable to capture. This limitation leads to a non-linear ML paradigm, that can be based on the “kernelization” of linear methods or purely non-linear mapping methods. The last cases lead, for the Euclidean distance, to a kernelized version combining the learned transformation $\varphi (\mathbf{x}):{\mathbb{R}}^m\to {\mathbb{R}}^{\overline{m}}$ with a Euclidean distance function with the capability to capture highly non-linear similarity relations, that is ${\mathcal{D}}_{ij}^{\varphi }=\parallel (\varphi ({\mathbf{x}}_i)-\varphi ({\mathbf{x}}_j)){\parallel }_2$ (Kedem et al., 2012). Local metric refers to a problem where multiple local metrics are learned and often relies on heterogeneous data objects. In the last setting, algorithms learn using only local pairwise constraints. According to the scheme depicted in Fig. 2 the scalability of the solution is a challenging task, especially if we consider the growing of the availability of data in the Big Data era. The scalability could be important under the dataset dimension n and/or the dimensionality of data m. Finally, the intrinsic optimization task underlying the ML paradigm makes the optimality of the solution another important aspect. The latter, depends on the structure of the optimization scheme, that is, if the problem is convex or not (Boyd & Vandenberghe, 2004). In fact, for convex formulations it is guaranteed to reach a global maximum. On the contrary, for non-convex formulations, the solution may only be a local optimum.

On metric spaces and dissimilarity matrices

The weighted euclidean distance

Let be $\mathbf{X}\subseteq {\mathbb{R}}^{m\times n}$ a m × n data matrix with n data objects, arranged as columns, ${\mathbf{x}}_i={\left[x_{1i},...,x_{mi}\right]}^T\in {\mathbb{R}}^m$, where m is the dimension of the vectorial space where data points lie. The vector space is endowed with the standard scalar product ${\mathbf{x}}_i\cdot {\mathbf{x}}_j=⟨{\mathbf{x}}_i,{\mathbf{x}}_j⟩={\mathbf{x}}_i^T{\mathbf{x}}_j={\sum }_{k=1}^m{x}_{ik}{x}_{jk}{e}_{ik}{e}_{jk}$ while ${\mathbf{e}}_i$ is the i-th standard basis vector, i.e., a vector of all zeros except for the entry k, which has a 1. The Euclidean distance function⁹ in such space equipped with the standard inner product $⟨\cdot ,\cdot ⟩$ can be expressed as:

(7) $$\begin{array}{cc}d({\mathbf{x}}_i,{\mathbf{x}}_j)\hfill & =\sqrt{{\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert }_2}=\sqrt{⟨{\mathbf{x}}_i-{\mathbf{x}}_j,{\mathbf{x}}_i-{\mathbf{x}}_j⟩}=\hfill \\ \hfill & =\sqrt{{\left({\mathbf{x}}_i-{\mathbf{x}}_j\right)}^T\left({\mathbf{x}}_i-{\mathbf{x}}_j\right)}=\sqrt{\sum _{k=1}^m{\left(x_{ki}-x_{kj}\right)}^2},\hfill \end{array}$$where the elements ${\mathcal{D}}_{ij}=d({\mathbf{x}}_i,{\mathbf{x}}_j),i,j=1,2,...,n$ form the entries of the n × n distance matrix $\mathcal{D}$ between the objects ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$ in $\mathbf{X}$.

Given a symmetric positive-definite matrix $\mathbf{M}$ with real-valued entries, i.e., $\mathbf{M}={\mathbf{M}}^T$ and ${\mathbf{x}}^T\mathbf{M}\mathbf{x}\ge 0$, $\mathbf{x}\ne 0$, the entry ${\mathcal{D}}_{ij}^{\mathbf{M}}$ of the WED matrix ${\mathcal{D}}^{\mathbf{M}}$ can be expressed as:

(8) $$\begin{array}{cc}{\mathcal{D}}_{ij}^{\mathbf{M}}=d_{\mathbf{M}}({\mathbf{x}}_i,{\mathbf{x}}_j)\hfill & =\sqrt{{\left({\mathbf{x}}_i-{\mathbf{x}}_j\right)}^T\mathbf{M}({\mathbf{x}}_i-{\mathbf{x}}_j)}=\sqrt{{({\mathbf{x}}_i-{\mathbf{x}}_j)}^T{\mathbf{W}}^T\mathbf{W}({\mathbf{x}}_i-{\mathbf{x}}_j)}=\hfill \\ \hfill & =\sqrt{{(\mathbf{W}({\mathbf{x}}_i-{\mathbf{x}}_j))}^T\mathbf{W}({\mathbf{x}}_i-{\mathbf{x}}_j)}=\hfill \\ \hfill & =\sqrt{⟨\mathbf{W}({\mathbf{x}}_i-{\mathbf{x}}_j),\mathbf{W}({\mathbf{x}}_i-{\mathbf{x}}_j)⟩}=\sqrt{\parallel \mathbf{W}({\mathbf{x}}_i-{\mathbf{x}}_j){\parallel }_2},\hfill \\ \hfill & \hfill \end{array}$$where $\mathbf{M}={\mathbf{W}}^T\mathbf{W}$ is the Cholesky decomposition (Strang, 1976) of matrix $\mathbf{M}$, that in the Hermitian general case, is found to be the decomposition of an Hermitian matrix in the product of a lower triangular matrix and its conjugate transpose.

In ML literature the distance in Eq. (8) is known as generalized Mahalanobis distance, a family of quadratic distances parametrized by a matrix $\mathbf{M}\in {\mathbb{S}}_{+}^m$, where ${\mathbb{S}}_{+}^m$ is the cone of symmetric positive semi-definite (PSD) m × m real-valued matrices–see Fig. 3. Note that $\mathbf{M}\in {\mathbb{S}}_{+}^m$ ensures that the function $d_{\mathbf{M}}$ satisfies the properties of a pseudo-distance, i.e., $\mathrm{\forall }\mathbf{x},\mathbf{y},\mathbf{z}\in \mathbf{X}$ holds:

1. $d_{\mathbf{M}}(\mathbf{x},\mathbf{y})\ge 0$ (non-negativity);

2. $d_{\mathbf{M}}(\mathbf{x},\mathbf{x})=0$ (identity);

3. $d_{\mathbf{M}}(\mathbf{x},\mathbf{y})=d_{\mathbf{M}}(\mathbf{y},\mathbf{x})$ (symmetry);

4. $d_{\mathbf{M}}(\mathbf{x},\mathbf{z})\le d_{\mathbf{M}}(\mathbf{x},\mathbf{y})+d_{\mathbf{M}}(\mathbf{y},\mathbf{z})$ (triangular inequality).

The above properties hold trivially for a standard Euclidean space where $\mathbf{M}=\mathbf{I}$.

The matrix $\mathbf{W}$ can be seen as a linear operator that transforms the shape of the space where data points, i.e., data vectors, lie. Specifically $\mathbf{W}$ defines a suitable transformation (endomorphism) $\mathcal{V}\to \mathcal{V}$ of the (abstract) space $\mathcal{V}$ spanned by rows vector of $\mathbf{X}$ in itself: given a vector $\mathbf{x}$ in the starting space S₁, the matrix $\mathbf{W}$ maps this vector in a new vector ${\mathbf{x}}_w=\mathbf{W}\mathbf{x}$ that lies in the space S₂, where S₁ and S₂ are isomorphic to $\mathcal{V}$¹⁰ . In the new transformed space the inner product becomes the standard inner product $⟨\cdot ,\cdot ⟩$, i.e., $⟨{\mathbf{x}}_w,{\mathbf{y}}_w⟩=⟨\mathbf{W}\mathbf{x},\mathbf{W}\mathbf{y}⟩={\mathbf{x}}_w^T{\mathbf{y}}_w=(\mathbf{W}\mathbf{x}{)}^T(\mathbf{W}\mathbf{y})={\mathbf{x}}^T{\mathbf{W}}^T\mathbf{W}\mathbf{x}$. The arrival space is endowed with a squared norm given by $⟨{\mathbf{x}}_w,{\mathbf{x}}_w⟩=⟨\mathbf{W}\mathbf{x},\mathbf{W}\mathbf{x}⟩={\Vert \mathbf{M}\mathbf{x}\Vert }_2^2$, being $\mathbf{M}={\mathbf{W}}^T\mathbf{W}$¹¹ .

Observation 1. The weighted distance $d_{\mathbf{M}}(\mathbf{x},\mathbf{y})=d(\mathbf{x},\mathbf{y},\mathbf{M})$ with $\mathbf{M}={\mathbf{W}}^T\mathbf{W}$ equals $d_{\mathbf{I}}({\mathbf{x}}_w,{\mathbf{y}}_w)=d({\mathbf{x}}_w,{\mathbf{y}}_w,\mathbf{I})$ where ${\mathbf{x}}_w=\mathbf{W}\mathbf{x}$ and ${\mathbf{y}}_w=\mathbf{W}\mathbf{y}$ and $\mathbf{I}$ is the identity matrix.

Proof. The proof follows by the same algebraic manipulation of Eq. (8).

Let $\mathbf{M}={\mathbf{W}}^T\mathbf{W}$, ${\mathbf{x}}_w=\mathbf{W}\mathbf{x}$ and ${\mathbf{y}}_w=\mathbf{W}\mathbf{y}$. It holds that:

(9) $$\begin{array}{cc}d\left(\mathbf{x},\mathbf{y},\mathbf{M}\right)\hfill & =\sqrt{{\left(\mathbf{x}-\mathbf{y}\right)}^T\mathbf{M}\left(\mathbf{x}-\mathbf{y}\right)}=\sqrt{{\left(\mathbf{x}-\mathbf{y}\right)}^T{\mathbf{W}}^T\mathbf{W}\left(\mathbf{x}-\mathbf{y}\right)}=\hfill \\ \hfill & =\sqrt{{\left(\mathbf{W}\left(\mathbf{x}-\mathbf{y}\right)\right)}^T\mathbf{W}\left(\mathbf{x}-\mathbf{y}\right)}=\sqrt{{\left(\mathbf{W}\mathbf{x}-\mathbf{W}\mathbf{y}\right)}^T\left(\mathbf{W}\mathbf{x}-\mathbf{W}\mathbf{y}\right)}=\hfill \\ \hfill & =\sqrt{{\left({\mathbf{x}}_w-{\mathbf{y}}_w\right)}^T\left({\mathbf{x}}_w-{\mathbf{y}}_w\right)}=d({\mathbf{x}}_w,{\mathbf{y}}_w,\mathbf{I}).\hfill \end{array}$$

□

The matrix $\mathbf{W}$ is an instance of an operator that defines a rotation and a scaling of the objects upon it operates. $\mathbf{W}$ maps a circle in the unweighted Euclidean space in an ellipse in the weighted Euclidean space–see Fig. 4. Hence we can state the following theorem.

Theorem 1. Applying a transformation $\mathbf{W}$ to all point of a circle of radius r the resulting points form an ellipse whose center is the same as the circle and length of its axes equals r times twice the square root of eigenvalues of $\mathbf{M}={\mathbf{W}}^T\mathbf{W}$.

Proof. An interesting demonstration can be found in Kurniawati, Jin & Shepherd (1998). □

The weight matrix $\mathbf{M}$ can be decomposed in its rotation and scaling components by means of the eigendecomposition operation. Specifically, by decomposing $\mathbf{M}=\mathbf{Q}\mathbf{D}{\mathbf{Q}}^T$ where $\mathbf{Q}$ is an orthogonal matrix with normalized column vectors, that is ${\mathbf{Q}}^T\mathbf{Q}=\mathbf{I}$ and ${\mathbf{Q}}^T={\mathbf{Q}}^{-1}$, and $\mathbf{D}$ is a diagonal matrix¹² . $\mathbf{D}$ contains the eigenvalues λ₁,λ₂,…,λ_m (organized in decreasing order) that are the scaling factors, while $\mathbf{Q}$ is the rotation operator matrix that leaves unchanged the (squared) norm of vectors, that is ${\Vert \mathbf{Q}\mathbf{x}\Vert }_2^2=⟨\mathbf{Q}\mathbf{x},\mathbf{Q}\mathbf{x}⟩={\mathbf{x}}^T{\mathbf{Q}}^T\mathbf{Q}\mathbf{x}={\mathbf{x}}^T\mathbf{x}={\Vert \mathbf{x}\Vert }_2^2$ (Strang, 1976).

At this point it is possible to express the WED in terms of the above eigendecomposition:

(10) $$\begin{array}{cc}d\left(\mathbf{x},\mathbf{y},\mathbf{M}\right)\hfill & =\sqrt{{\left(\mathbf{x}-\mathbf{y}\right)}^T\mathbf{M}\left(\mathbf{x}-\mathbf{y}\right)}=\sqrt{{\left(\mathbf{x}-\mathbf{y}\right)}^T{\mathbf{W}}^T\mathbf{W}\left(\mathbf{x}-\mathbf{y}\right)}=\hfill \\ \hfill & =\sqrt{{\left(\mathbf{x}-\mathbf{y}\right)}^T\mathbf{Q}\mathbf{D}{\mathbf{Q}}^T\left(\mathbf{x}-\mathbf{y}\right)}=\sqrt{{\left({\mathbf{Q}}^T\mathbf{x}-{\mathbf{Q}}^T\mathbf{y}\right)}^T\mathbf{D}\left({\mathbf{Q}}^T\mathbf{x}-{\mathbf{Q}}^T\mathbf{y}\right)}.\hfill \end{array}$$

From Eq. (10) it follows that $d\left(\mathbf{x},\mathbf{y},\mathbf{M}\right)$ can be expressed, through the eigendecomposition of the weighting matrix $\mathbf{M}$, with another weighted distance with weights given by the eigenvalues matrix $\mathbf{D}$. This new distance takes into account new vectors: ${\mathbf{Q}}^T\mathbf{x}=\hat{\mathbf{x}}$ and ${\mathbf{Q}}^T\mathbf{y}=\hat{\mathbf{y}}$ that are the rotated counterparts of original vectors $\mathbf{x}$ and $\mathbf{y}$. In other words, the two vectors $\hat{\mathbf{x}}$ and $\hat{\mathbf{y}}$ are the rotated, but not scaled, version of ${\mathbf{x}}_w$ and ${\mathbf{y}}_w$ that originate both in space S₂. It can be demonstrated that the length of the axis of the ellipsoid in the direction of i-th eigenvalue λ_i is equal to: $\sqrt{d\left(\mathbf{x},\mathbf{y},\mathbf{M}\right)/{\lambda }_i}$.

Finally if the weighting matrix $\mathbf{M}$ is a diagonal matrix, with real entries, the above eigendecomposition reduces to $\mathbf{M}=\mathbf{E}\mathbf{D}{\mathbf{E}}^T$ where $\mathbf{E}$ is the eigenvector matrix $\mathbf{E}=\left[{\mathbf{e}}_1,{\mathbf{e}}_2,...,{\mathbf{e}}_m\right]$, whose columns contains the standard basis in ${\mathbb{R}}^m$ with the property ${\mathbf{e}}_i^T{\mathbf{e}}_j={\delta }_{ij}$, where δ_ij is the Kronecker delta. In this case the matrix $\mathbf{E}$ represents the identity element of the rotation operator, leaving vectors in the original place, while they are scaled by a factors given by the entries of the diagonal of $\mathbf{M}$, being the eigenvalues of a diagonal matrix the diagonal entries of the same matrix. An example of this phenomenon is given in Fig. 5.

Characterization of a composite component-wise dissimilarity

When the PR problem at hand deals with heterogeneous measures on objects and these measures are both structurally and semantically different (graphs, time series, images, real numbers, etc.), a composite dissimilarity measure can be useful, for example in clustering applications. The dissimilarity measure is a combination of (sub)-dissimilarities suitably defined depending on the nature of the data.

Before constructing a toy composite dissimilarity measure, it is worth to mention the following corollary valid when a dissimilarity measure is computed by combining the dissimilarities pertaining to all of the m attributes separately. In fact, given m features, a dissimilarity measures can be computed as: $d(x,y)={\sum }_{i=1}^{m}f(x_i,y_i)$, where f (x_j, x_j) = 0 and f (x_j, y_j) = f (y_j, x_j) ≥ 0 for all j. The corollary states:

Corollary 1. Let $x,y\in {\mathbb{R}}^m$. Then $d(x,y)={\sum }_{i=1}^{m}f(x_i,y_i)$ is metric iff f is metric on $\mathbb{R}$.

Proof. The proof can be done considering that f is non-negative, symmetric and it holds that f (s, s) = 0 for $s\in \mathbb{R}$, then the first three axioms about metric spaces, i.e., reflexivity, symmetry and definiteness, are fulfilled. Furthermore, since d is metric d(x, y) + d(y, z) ≥ d(x, z) holds for all x, y, z. If we consider x_j = c_x, y_j = c_y, z_j = z_x, for all j and some constants c_x, c_y, c_j the Triangle inequality for d reduces to f(x_c, y_c) + f(y_c, z_c) ≥ f(x_c, z_c). The ⇒ proof is trivial. □

Moreover, it can be demonstrated (Gower & Legendre, 1986) that, at least for the Euclidean case (p = 2 in the Minkowski distance definition), if $f:X\times Y\to {\mathbb{R}}_{+}^0$ is a function, then $d(x,y)={\sum }_{i=1}^{m}f(x_i,y_i)$ is metric iff $d^{{}^{\prime }}(x,y)={\sum }_{i=1}^m{\left[f{(x_i,y_i)}^2\right]}^{1/2}$ is metric.

Now we show a demonstration of the following claim valid for a composite dissimilarity measure, making use of Def. 3 that characterizes the Euclidean behavior for dissimilarity matrices and Def. 2 for metric behavior.

Claim 1. Given two general objects $x,y\in \mathcal{H}$, where $\mathcal{H}$ is a generic feature space, and a component wise custom-based dissimilarity $d(x,y)=\sqrt{{(x\ominus y)}^T(x\ominus y)}=\sqrt{{\sum }_{i=1}^m{(x_i\ominus y_i)}^2}$, then if at least one component-wise dissimilarity is not Euclidean the dissimilarity matrix that arises from d applied on object within $\mathcal{H}$, is not Euclidean.

As stated in “On Metric Spaces and Dissimilarity Matrices”, the expression “non-Euclidean” means that there is no set of vectors in a vector space of any dimensionality for which the Euclidean distances between the objects are identical to the given ones (Duin & Pękalska, 2010). We show now how Claim 1 can be demonstrated with a constructive example. Let $\mathbf{x}=(x_1,x_2,...x_k,x_{k+1},...x_m)$ and $\mathbf{y}=(y_1,y_2,...y_k,y_{k+1},...y_m)$ be two objects in a vectorial space ${\mathcal{H}}_v$. We define a set of component-wise dissimilarities induced for the first k components such as $f_j^{cw}(x_j,y_j)=\left|x_j-y_j\right|,j=1,2,...,k$ and a single component-wise dissimilarity induced for the remaining m − k components such as $f^p(x^{s=k+1,...,m},y^{s=k+1,...,m})={\left({\sum }_{s=k+1}^m{\left|x_s-y_s\right|}^p\right)}^{{\displaystyle \frac{1}{p}}}$. In other words we divide the starting space ${\mathcal{H}}_v$ as the Cartesian product (Strang, 1976) between two sub-spaces, the space ${\mathcal{H}}_{cw}$ generated from the first k components, in which the component-wise dissimilarities are computed as f^cw and ${\mathcal{H}}_p$ where the dissimilarity is computed as the Minkowski distance f^p applied to the last m − k components. Finally, the overall dissimilarity between two objects, say $\mathbf{x},\mathbf{y}$ is induced by the ℓ₂ norm in the following way:

(11) $$\hat{d}(\mathbf{x},\mathbf{y})=\sqrt{{\left(f^{cw}\oplus f^p\right)}^T\left(f^{cw}\oplus f^p\right)},$$where (f^cw ⊕ f^p) is the vector of dimension k + l constructed by the concatenation of the two (sub)-dissimilarities f^cw_j, f^p, j = 1,2,…,k.

To evaluate the validity of the Claim 1 the dissimilarity in Eq. (11) is computed on a sample drawn from a multi-variate Gaussian distribution with dimension m parameterized as: m = k + l, where k is maintained fixed without loss of generality, and l is varied. It is noted that the p parameter controls the nature of the Minkowski distance, making the (sub)-dissimilarity f^p metric or not metric (and even non-Euclidean¹³ ) depending on the value of p as demonstrated above, such that for p ≥ 1 it is metric.

In order to measure the non-Euclidean behavior of the space induced by the Minkowski distance, we introduce the Negative Eigen-Fraction (NEF):

(12) $$NEF={\displaystyle \frac{\sum _{j=p+1}^{p+q}\left|{\lambda }_i\right|}{\sum _{i=1}^{p+q}\left|{\lambda }_j\right|},}$$where (p, q) is the signature of the PE space, and λ_i are the eigenvalues of the Gram matrix decomposition. The NEF measures the degree of the non-Euclidean influence evaluating the ratio between the sum of the negative eigenvalues and the overall set of eigenvalues. Another index that helps to commensurate the non-Euclidean influence is the Negative Eigen-Ratio (NER):

(13) $$NER=r_1={\displaystyle \frac{\left|{\lambda }_{min}\right|}{{\lambda }_{max}},}$$where λ_min and λ_max are the minimum and maximum eigenvalue of the Gram matrix. In Fig. 6 are reported, following the same experimental scheme proposed in Pękalska et al. (2006), several curves representing the NEF for a 100 points Gaussian sample varying the p parameter of the Minkowski distance as a function of the dimensionality. Now it is clear that the Minkowski distance is non-Euclidean for any p ≠ 2, but for very high dimensionality values the Euclidean behavior is restored independently from p.

However, from Def. 3, we know that a n × n (n ≥ m) dissimilarity (distance) matrix $\mathcal{D}$ is Euclidean if it can be embedded in a Euclidean space $\left({\mathbb{R}}^m,d_2\right)$, where d₂ is the standard Euclidean distance. It means that the Gram matrix $\mathbf{G}$ obtained as described in “On Metric Spaces and Dissimilarity Matrices” does not contain negative eigenvalues, hence it is a positive semi-definite matrix. The remainder of the discussion is then based on the eigenvalues spectrum of the Gram matrix computed from the dissimilarity matrix $\hat{\mathcal{D}}={\hat{d}}_{ij}$. In Fig. 7 are reported the eigenvalues spectra for the Gram matrix $\hat{\mathbf{G}}$ obtained from the dissimilarity matrix $\hat{\mathcal{D}}$ computed for a fixed k = 5 and varying the value for l = 0, 1,… 4. The dashed lines are the case: l = 0 and l = 1. The first one represents the spectrum deducted from the first k = 5 components of ${\mathcal{H}}_v$ and, as we expected, it contains only positive eigenvalues, thereby the dissimilarity matrix $\hat{\mathcal{D}}$ is isometrically embeddable. The same holds for l = 1 because trivially we have that $f_p(x,y)={\left({\left|x-y\right|}^p\right)}^{{\displaystyle \frac{1}{p}}}=\left|x-y\right|,\mathrm{\forall }x,y\in \mathbb{R}$, thus the dissimilarity measure f_p remains metric. For l >1 the several spectra contain both positive and negative eigenvalues making the Gram matrix $\hat{\mathbf{G}}$ indefinite. As counterexample in Fig. 8 are depicted the spectra of the dissimilarity in Eq. (11), where the parameter p of the Minkowski distance is set as p = 2. As we expect, in this case, the dissimilarity behaves in an Euclidean fashion.

On the presence of weights in a component-wise dissimilarity and the eigenspectrum of the gram matrix

In the discussion related to Claim 1 it is introduced a suitable component-wise custom dissimilarity that in general has the form: $d={\Vert {\overline{\mathbf{d}}}_c\Vert }_2=\sqrt{{\overline{\mathbf{d}}}_c^T{\overline{\mathbf{d}}}_c}$ that is the $\mathbf{\ell }$₂ norm of the vector¹⁴ ${\overline{\mathbf{d}}}_c=\left[d_{{\mathcal{F}}_1},d_{{\mathcal{F}}_2},...,d_{{\mathcal{F}}_k}\right]$ that is computed through suitable component-wise (sub)-dissimilarities, each one induced for a specific feature type ${\mathcal{F}}_j$. Specifically, we had two groups: the first (sub)-dissimilarities act on a vectorial subspace and they were computed as the component-wise $\mathbf{\ell }$₁ norm: $\left|x_i-y_i\right|$, while in the second group a unique (sub)-dissimilarity is computed as the Minkowski distance (p = 0.8, hence neither metric, nor Euclidean). Now we will discuss the case in which the same family of custom-based dissimilarities are weighted, hence they have the form described in “The Weighted Euclidean Distance” for the WED. In other words, given a pair of objects x_i and y_j, the dissimilarity measure under analysis has the following form:

(14) $$d_{\mathbf{w}}\left(x_i,y_j\right)={\Vert {\overline{\mathbf{d}}}_c\Vert }_{\mathbf{w}}=\sqrt{{\overline{\mathbf{d}}}_c{\left(x_i,y_j\right)}^T{\mathbf{W}}^T\mathbf{W}{\overline{\mathbf{d}}}_c\left(x_i,y_j\right)}.$$

Given n objects x_i, i = 1,2,…,n, the weighted dissimilarity matrix whose entries are given by $d_{\mathbf{w}}\left(x_i,y_j\right)$–see Eq. (14)–is hereinafter referred to as ${\mathcal{D}}_{\mathbf{w}}$, for convenience. The latter can be decomposed according to Eq. (1) as ${\mathbf{G}}_{\mathbf{w}}=-{\displaystyle \frac{1}{2}{\mathcal{J}\mathcal{D}}_{\mathbf{w}}^{\ast 2}\mathcal{J}}$, where ${\mathbf{G}}_{\mathbf{w}}$ is the Gram matrix parametrized by the weight matrix $\mathbf{W}$. As discussed in “Metric Learning”, the weights act as a linear mapping $\mathcal{M}:\mathbf{x}\to \mathbf{W}\mathbf{x}$. Starting from the above settings, two questions arise. The first is if, in principle, it is possible to find a suitable weighting matrix $\mathbf{W}$ that makes the dissimilarity matrix ${\mathcal{D}}_{\mathbf{w}}$ “more Euclidean”. The second question is about the behavior of the Gram matrix ${\mathbf{G}}_{\mathbf{w}}$ in terms of eigendecomposition. In other words, one may ask what is the relationship between the eigenvalues (and eigenvectors) of the non-weighted Gram matrix $\mathbf{G}$ and the weighted one ${\mathbf{G}}_{\mathbf{w}}$.

The two questions are strongly interrelated. By the way, the first is simpler than the second. To answer the first question one may conceive a simple problem in which one wants to minimize the NEF defined in Eq. (12), hence, we can consider a diagonal matrix ${\mathbf{W}}_{diag}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(\left[w_1,w_2,...,w_d\right])$ and the task is to solve the following minimization problem:

(15) $$\begin{array}{c}\hfill argmi{n}_{{\mathbf{G}}_{\mathbf{w}}}\mathrm{N}\mathrm{E}\mathrm{F}({\mathbf{G}}_{\mathbf{w}}),\hfill \\ \hfill {\text{G}}_{\text{W}}\hfill \\ \hfill s.t.0\le w_i\le 1i=1,2,...,d.\hfill \end{array}$$

The NEF – see Eq. (12) – depends on the eigenvalues λ_i of the Gram matrix which, in turn, depend on the dissimilarity matrix ${\mathcal{D}}_{\mathbf{w}}$ through a non-linear operation, which in turn depends on the weighted dissimilarity measure $d_{\mathbf{w}}\left(x_i,y_j\right)$, which, finally, depends on the weights matrix ${\mathbf{W}}_{diag}$ (if diagonal). The optimization problem can be performed via the same setting used to discuss Claim 1. Specifically, it is a simple exercise in adopting a meta-heuristic, such as a GA, in minimizing the optimization problem in Eq. (15). The two subspaces, ${\mathcal{H}}_{cw}$ and ${\mathcal{H}}_v$ have a dimensionality equal to 3 and the Minkowski parameter of the distance acting on ${\mathcal{H}}_{cw}$ is set to 0.8 (hence neither metric, nor Euclidean).

Starting from a random population of 30 individuals (chromosomes) for the weights $\mathbf{w}$, the GA converges to the (sub)-optimal solution ${\mathbf{w}}^{\ast }=\left[1,1,0.999,0.0001\right]$ with a fitness value (the NEF) equals to 2.0380e-06, hence negligible. As we expected, the GA finds a solution with higher weights for the “Euclidean” components and practically null value for the “Minkowski” component.

Although the answer to the first question is trivial, the second question about the relationship of the two spectra of $\mathbf{G}$ and ${\mathbf{G}}_{\mathbf{w}}$ is only apparently simple. Here we try to give a sketch of the problem. Suppose that $\mathcal{F}$ is a vectorial space endowed with the standard norm $⟨\cdot ,\cdot ⟩$, and $\mathbf{X}\in {\mathbb{R}}^{m\times n}$ is a data matrix with the n data points organized as columns. The discussion can be restricted to an Euclidean space equipped by the standard Euclidean distance: $d\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)={\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert }_2$ for ${\mathbf{x}}_i,{\mathbf{x}}_j\in \mathbf{X}$. The scalar product matrix or the Gram matrix, with the data matrix organized with data vectors in columns and the variables as rows, is: $\mathbf{G}={\mathbf{X}}^T\mathbf{X}$. The linear mapping $\mathcal{M}:\mathbf{x}\to \mathbf{W}\mathbf{x}$ transforms the data matrix $\mathbf{X}$ in $\mathcal{M}\left(\mathbf{X}\right)=\mathbf{W}\mathbf{X}=\mathbf{Y}$. Thereby, the Gram matrix becomes: ${\mathbf{G}}_{\mathbf{w}}={\left(\mathbf{W}\mathbf{X}\right)}^T\left(\mathbf{W}\mathbf{X}\right)={\mathbf{X}}^T{\mathbf{W}}^T\mathbf{W}\mathbf{X}={\mathbf{Y}}^T\mathbf{Y}$. We note that if $\mathbf{W}$ is invertible we have the inverse map ${\mathcal{M}}^{-1}\left(\mathbf{Y}\right)={\mathbf{W}}^{-1}\mathbf{Y}$. In trying to find a relation between the eigenvalues of ${\mathbf{Y}}^T\mathbf{Y}$ and those of ${\mathbf{X}}^T\mathbf{X}$, we can make use of the relation between the Singular Value Decomposition (SVD) of a m × n matrix $\mathbf{A}$ and the eigendecomposition of the n × n matrix ${\mathbf{A}}^T\mathbf{A}$. In fact, any m × n matrix can be factored as $\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^T$ (Strang, 1976), where the columns of matrix $\mathbf{U}$ (m × m) are the eigenvectors of $\mathbf{A}{\mathbf{A}}^T$ and the columns of $\mathbf{V}$ (n × n) are eigenvectors of ${\mathbf{A}}^T\mathbf{A}$; finally, the $r=rank\left(\mathbf{A}\right)$ singular values in the diagonal of Σ (m × n) are the square roots of the non-zero eigenvalues of both ${\mathbf{A}}^T\mathbf{A}$ and $\mathbf{A}{\mathbf{A}}^T$¹⁵ .

Let $\mathbf{Y}={\mathbf{U}}_{\mathbf{w}}{\mathrm{\Sigma }}_{\mathbf{w}}{\mathbf{V}}_{\mathbf{w}}^T$ be the SVD decomposition of $\mathbf{Y}$ and $\mathbf{X}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^T$ be the decomposition of $\mathbf{X}$. If we multiply on the left side for ${\mathbf{W}}^{-1}$ both sides of the first relation we obtain ${\mathbf{W}}^{-1}\mathbf{Y}={\mathbf{W}}^{-1}{\mathbf{U}}_{\mathbf{w}}{\mathrm{\Sigma }}_{\mathbf{w}}{\mathbf{V}}_{\mathbf{w}}^T$, hence $\mathbf{X}={\mathbf{W}}^{-1}\mathbf{Y}={\mathbf{W}}^{-1}{\mathbf{U}}_{\mathbf{w}}{\mathrm{\Sigma }}_{\mathbf{w}}{\mathbf{V}}_{\mathbf{w}}^T$. If we compare these two relations and multiply both sides for ${\mathbf{U}}^T$ on the left side and $\mathbf{V}$ on the right side and by further considering that ${\mathbf{V}}^T\mathbf{V}=\mathbf{I}={\mathbf{U}}^T\mathbf{U}$, we come to the relation: ${\mathbf{U}}^T\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^T\mathbf{V}={\mathbf{U}}^T{\mathbf{W}}^{-1}{\mathbf{U}}_{\mathbf{w}}{\mathrm{\Sigma }}_{\mathbf{w}}{\mathbf{V}}_{\mathbf{w}}^T\mathbf{V}$ that simplifies as:

(16) $$\mathrm{\Sigma }={\mathbf{U}}^T{\mathbf{W}}^{-1}{\mathbf{U}}_{\mathbf{w}}{\mathrm{\Sigma }}_{\mathbf{w}}{\mathbf{V}}_{\mathbf{w}}^T\mathbf{V}.$$

Equation (16) is a (complex) relation between the (diagonal) singular values matrix Σ that contains as entries the singular values of $\mathbf{X}$ and the singular values of $\mathbf{Y}=\mathbf{W}\mathbf{X}$, placed in the diagonal of ${\mathrm{\Sigma }}_{\mathbf{w}}$. Unfortunately, calculations cannot be further performed in closed form unless we make additional assumption on $\mathbf{W}$. The reason becomes clear if we think at $\mathbf{W}\mathbf{X}$ as the product of two matrices: in fact, the original question about the relationship between the eigenvalues of the Gram matrices $\mathbf{G}$ and ${\mathbf{G}}_{\mathbf{w}}$ can be translated into the relation of the eigenvalues of the following matrices $\mathbf{A},\mathbf{B},\mathbf{A}\mathbf{B}$. However, this so-wanted relationship between the eigenvalues of the product of general matrices and its multiplicands is still an open problem of mathematics, even if in the literature there are a number of works that provide several inequalities for the matrix product and sum problem (Zhang & Zhang, 2006; Fulton, 2000; Watkins, 1970; Thompson & Therianos, 1971). If $\mathbf{W}$ is a scalar matrix of the form $\mathbf{W}=k\mathbf{I}$ the relation shown in Eq. (16) becomes simple. In fact, we can write $\mathbf{W}\mathbf{X}=k\mathbf{I}\mathbf{X}={\mathbf{U}}_{\mathbf{w}}{\mathrm{\Sigma }}_{\mathbf{w}}{\mathbf{V}}_{\mathbf{w}}^T$, but $\mathbf{X}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^T$, hence $\mathbf{W}\mathbf{X}=k\mathbf{I}\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^T={\mathbf{U}}_{\mathbf{w}}{\mathrm{\Sigma }}_{\mathbf{w}}{\mathbf{V}}_{\mathbf{w}}^T$. It means that the singular vectors are the same: $\mathbf{U}={\mathbf{U}}_{\mathbf{w}}$ and $\mathbf{V}={\mathbf{V}}_{\mathbf{w}}$ and therefore Eq. (16) becomes:

(17) $$\mathrm{\Sigma }=k^{-1}{\mathrm{\Sigma }}_{\mathbf{w}}\Rightarrow {\mathrm{\Sigma }}_{\mathbf{w}}=k\mathrm{\Sigma },$$

Hence, for the spectrum of ${\mathbf{G}}_{\mathbf{w}}={\mathbf{X}}^T{\mathbf{W}}^T\mathbf{W}\mathbf{X}$, we have ${\mathrm{\Sigma }}^T\mathrm{\Sigma }=k^{-1}{\mathrm{\Sigma }}_{\mathbf{w}}^T{\mathrm{\Sigma }}_{\mathbf{w}}$¹⁶ .

Ultimately, there are no relationships between the spectrum of the product of two generic matrices and one of the single matrices, unless in simple cases¹⁷ . In general, two generic matrices do not share the same set of eigenvectors and this makes the analysis infeasible. In order to graphically show in a computational fashion the relationship between the eigenvalues of the Gram matrix obtained from a weighted dissimilarity matrix and those obtained from a non-weighted dissimilarity matrix, we have generated a random bi-dimensional matrix ${\mathbf{X}}^{test}\in {\mathbb{R}}^{(2\times 20)}$, hence containing 20 random 2-D vectors. Moreover, the dissimilarity matrix ${\mathcal{D}}_{\mathbf{w}}^{test}$ on ${\mathbf{X}}^{test}$ is computed through the standard Euclidean distance and finally the Gram matrix ${\mathbf{G}}_{\mathbf{w}}^{test}$ is extracted. The dissimilarity measure is weighted with a diagonal matrix of the form: $\mathbf{W}=\left[\begin{array}{cc}\alpha & 0\\ 0& \beta \\ & \end{array}\right]$, where $\alpha ,\beta \in \left(0,1\right]$¹⁸ . Finally the eigendecomposition of ${\mathbf{G}}_{\mathbf{w}}^{test}$ is performed, yielding the first two eigenvalues ${\lambda }_{\mathbf{w}1}$ and ${\lambda }_{\mathbf{w}2}$ as function of $\mathbf{W}$.

In Fig. 9 are depicted the value of the first and the second eigenvalues of ${\mathbf{G}}_{\mathbf{w}}^{test}$, respectively, as a function of α and β in the predefined interval. In Fig. 10, as instead, it is reported the value of the first and second eigenvalues in the case α = β = k, that is the case $\mathbf{W}=k\mathbf{I}$.

For completeness in Fig. 11 are reported the sum, the product, and the quotient of the first two eigenvalues of ${\mathbf{G}}_{\mathbf{w}}^{test}$, while in Fig. 12 we have the same operations in the case of α = β = k.

Conclusion

In solving real-world problems in pattern recognition we may incur in a complex representation of objects with the need of a custom-based dissimilarity measure whose components are (sub)-dissimilarities tailored on the nature of the object at hand. Moreover, the starting space can be non-metric and standard machine learning algorithms cannot operate directly due to the absence of a vectorial space endowed with some well-defined norm. The dissimilarity template can be a weighted Euclidean distance where weights are learned by exploiting a metric learning paradigm. Often, in real-world applications, the adopted custom-based dissimilarity measure leads to non-Euclidean dissimilarity matrices. The non-Euclidean behavior can be suitably measured by studying the spectrum of the related Gram matrix. The adopted framework shows how the (sub)-dissimilarity measure adopted can affect the Euclidean behavior and how a weighting scheme can suitably address this phenomenon. The weighting scheme concerns the spectra of the underlying dissimilarity, but only in some simple cases the problem can be addressed theoretically. Alongside the present work of a more theoretical nature, as regards the future directions, we have planned to evaluate the impact of the non-metricity of the dissimilarity matrices in some real-world applications (e.g., predictive maintenance) and as a correction expressed directly in the objective function (in line with our theoretical discussion) of an optimization system impacts on the performance of a classification system in terms of generalization capabilities.