A novel autoencoder approach to feature extraction with linear separability for high-dimensional data

Theory

Given a sample X ={x_i — 1 ≤ i ≤ N}, and X⊆ℜ^d.ℜ^d is the d-dimensional Euclidean space. P is the probability distribution of X, denoted as original probability distribution. u(X) and Γ_x are the mean vector and the covariance matrix of X, respectively. Let us assume that Z ={ z_j — 1 ≤ j ≤ N} is the reconstructed X, and Z⊆ℜ^d. Q is the probability distribution of Z, denoted as approximate probability distribution. Similar, u( Z) and Γ_zare the mean vector and the covariance matrix of Z, respectively. The K–L divergence (Tao et al., 2009) between the two distributions P and Q is given in Eq. (1). (1)${\displaystyle K\left(P\left|\right|Q\right)=\frac{1}{2}\left[log\left|{\Gamma }_z\right|-log\left|{\Gamma }_x\right|+\mathrm{tr}\left({\Gamma }_z^{-1}{\Gamma }_x\right)+\mathrm{tr}\left({\Gamma }_z^{-1}{D}_{xz}\right)\right].}$ where |Γ| = det(Γ). The tr(⋅) is the trace of a matrix. D_xz = (u(X) − u(Z))(u(X) − u(Z))^T is a symmetrical matrix. Training a distance metric is equivalent to finding a rescaling of a sample which replaces each x_i with M^T x_i (Feng, Wang & Jin, 2019), so the K-L divergence in Eq. (1) can be converted into Eq. (2), having (2)${\displaystyle K_{{}_L}^{\mathrm{\ast }}\left(P\left|\right|Q\right)=\frac{1}{2}\left[log\left|M^T{\Gamma }_{z}M\right|-log\left|M^T{\Gamma }_{x}M\right|+\mathrm{tr}\left({\left(M^T{\Gamma }_{z}M\right)}^{-1}\left(M^T\left({\Gamma }_x+D_{xz}\right)M\right)\right)\right].}$ where M is a metric matrix and satisfies A∗ = MM^T, and M ∈ ℜ^d×d₀, d₀ ≤ d. The K-L divergence in Eq. (2) is the rescaling transformation for the K-L divergence in Eq. (1) using the distance metric matrix A *. To reduce the difference between the approximate distribution Q and the original distribution P, we consider Mahalanobis distance metric for K-L divergence in Eq. (2), having (3)${\displaystyle K-L\left(d_{A\ast }\right)=K_{{}_L}^{\mathrm{\ast }}\left(P\left|\right|Q\right)+\sum _{1\le i,j}{d}_{A\ast }\left(x_i,z_j\right).}$ d_A∗(x_i, z_j) is Mahalanobis distance between x_i and z_j using A *. The advantage of doing this is that the Mahalanobis distance using A * can appropriately measure similarities between the input sample and the reconstructed input sample because of non-negativity (i.e., d_A∗(x_i, z_j) ≥ 0), distinguishability (i.e., d_A∗(x_i, z_j) = 0⇔x_i = z_j) and symmetry (i.e., d_A∗(x_i, z_j) = d_A∗(z_j, x_i)) (Feng, Wang & Jin, 2019). Equation (4) gives the calculation of d_A∗(x_i, z_j), where A * can be decomposed as A∗ = MM^T. (4)${\displaystyle d_{A\ast }\left(x_i,z_j\right)=\sqrt{{\left(x_i-z_j\right)}^{T}A\ast \left(x_i-z_j\right)}.}$

Model implementation

A classic auto encoder (AE) consists of an input layer, a hidden-layer and an output layer. For AE, the loss error is often measured by using the distance between the original input instance, the predicted instances, and the reconstructed instance (Theis, Shi & Cunningham, 2017). Typically, using divergence metrics or expanding autoencoder structures (e.g., enlarging the number of hidden layers) is more helpful for autoencoders to characterize the data distribution and to learn the desired representations (Lu, Cheng & Xiao, 2017). As such, we designed an autoencoder with multiple-hidden layers, namely m-AE, and m ≥1, as shown in Fig. 1. In addition, the K–L divergence in Eq. (3) was used to increase the ability of m-AE to capture low-dimensional feature representations. The loss error ∇_KL(w, b) in m-AE is given as follows: (5)${\displaystyle {\nabla }_{KL}\left(\mathbf{w},\mathbf{b}\right)=\sum \left|\right|e_X-e_Z|{|}^2+K-L\left(d_{A\ast }\right).}$ where e_X, e_Z are the inputting and the reconstructed inputting, respectively. ∇_KL(w, b) isupdated through using the backpropagation manner.

To better train the proposed model, we carefully studied part hyper parameters in the model. For the rest of hyper parameters, their default values were used.

(i) Optimizer. Common optimizers are Adam, RMSprop, SGD, Momentum, Nesterov, etc. However, we selected Adam as the optimizer of m-AE, since Adam has the ability to handle sparse gradients (Kingma & Ba, 2015). Compared with other optimizers, Adam is more suitable for high-dimensional data. Moreover, Adam can provide different adaptive learning rates for different hyper parameters.

(ii) Activation function. Gradient vanishing is easily to be induced during passing gradients backwards for neural networks, in this case, the probability of gradient vanishing caused by activation function Sigmoid is relatively high. Similar to Sigmoid, activation function tanh also suffers from this problem. While for activation function ReLu, the phenomenon of gradient vanishing is partially alleviated, meaning that gradient vanishing does not appear in the positive interval of ReLu. Furthermore, ReLu converges much faster than Sigmoid and Tanh. Therefore, we chose ReLu as the activation function of m-AE.

(iii) Iteration epoch. We dynamically adjust the iteration epoch according to training accuracy. For instance, when training accuracy starts to change from large to small, we reduce iteration epoch in order to prevent over-fitting. When the difference in accuracy between training and testing is minimal, the current iteration epoch can be accepted and training procedure is stopped.

We give the training algorithm for m-AE in Algorithm 1. In the algorithm, the training set Train_set is divided into two datasets T^Cro_train, T^Cro_val in step 1. Since m-AE has multiple hidden layers, we set m in the range of O_m, in order to determine the m, the dataset T^Cro_train is used to train m-AE. The data set T^Cro_val is used for the validation of the network structure of m-AE. To get the optimal m, denoted as m_opt, the cross-validation is implemented in step 2 to step 18, where the procedure of step 6 and step 10 describes the calculation process of loss error ∇_KL(w, b). After gaining the optimal m, m-AE is trained using the training set Train_set. Using backpropagation manner updates network parameters until m-AE can converge, as shown in step 18 to step 28. The procedure shown in step 29 to step 33 indicates that the maximum training accuracy Train_acc are outputted and the well trained m-AE is saved.

Algorithm 1. Training for m-AE.

Input: Training set Train_set, A∗ = I ∈ ℜ^d×d is an identity matrix, iteration epoch T, L, parameter O_m.

Output: Training accuracy Train_acc.

Begin

1. Train_set is divided into T^Cro_train, T^Cro_val;

2. for t =1 to T do:

3. foreach m in O_m:

4. Decompose A * as satisfying A∗ = MM^T using eigen decomposition.

5. Calculate loss error ∇_KL(w, b) using Eq. (5) and the procedure is summarized as following:

6. The procedure:

7. Calculate $K_{{}_L}^{\mathrm{\ast }}\left(P\left|\right|Q\right)$ using Eq. (2).

8. Calculate d_A∗(x_i, z_j) using Eq. (4).

9. Take $K_{{}_L}^{\mathrm{\ast }}\left(P\left|\right|Q\right)$ and d_A∗(x_i, z_j) into Eq. (3) to calculate K − L(d_A∗).

10. For any x_i, x_j, calculate ∇_KL(w, b) using Eq. (5).

11. Calculate training accuracy T_acc = m − AE(T^Cro_train; m; t);

12. Validate m-AE using data set T^Cro_val;

13. Calculate validation accuracy V_acc = m − AE(T^Cro_val; m; t)

14. Update weight w←w + ∇w.

15. Update A * as MM^T.

16. Until A * and hyper parameters converge.

17. end foreach

18. end for

19. Get the optimal value of m, i.e., m_opt = arg max(V_acc);

20. for l =1 to L do:

21. Decompose A * as satisfying A∗ = MM^T using eigen decomposition.

22. Train m-AE using training set Train_set and m_opt;

23. Update network parameters using the optimizer Adam;

24. Calculate loss error ∇_KL(w, b) using Eq. (5);

25. Calculate training accuracy Training_acc(l) = m − AE(Train_set; m_opt);

26. Update A * as MM^T;

27. Using backpropagation manner updates network parameters;

28. end for

29. Select the l so that l_max =arg max(Training_acc(l));

30. Get the maximum training accuracy Train_acc in the l_max-th iteration;

31. Train_acc =m-AE(Train_set; m_opt, l_max);

32. Output Train_acc

33. Save the well trained m-AE(Train_set; m_opt, l_max);

End

Datasets and assessment metrics

To verify the performance of the proposed m-AE, we selected four benchmark datasets with different data dimensions from the UCI machine learning repository (Blake & Merz, 1998). The attributes of the four benchmark datasets are summarized in Table 1.

Receiver operating characteristic curve (ROC) and corresponding area under curve (AUC) are usually used to assess the precision of machine learning methods. Therefore, AUC is taken as the assessment metric of method precision.

Competing and benchmark methods

Since m-AE applies the distance metric of rescaling transformation, the methods based on a distance metric were used for comparisons, including ISSML (Ying, Wen & Shi, 2018) and ITML (Mei, Liu & Karimi, 2014). Certainly, the method based on feature selection was also considered, i.e., MMD-ISOMAP (Bo, Xiang & Zhang, 2016). In addition, autoencoder-based approaches were used as a comparison, e.g., the SAE (Chen, Hu & He, 2018). Furthermore, to further examine the effects of the distance metric of rescaling transformation on the performance of m-AE, a benchmark model was developed with m-AE as a reference. The developed benchmark model used the same structure and parameter configuration of m-AE without using the distance metric of rescaling transformation, namely AE-BK.

We implemented the corresponding algorithms of the six models using Python on Tensorflow framework. While for those parameters of competing methods, we adopted those values observed in the corresponding literature. Certainly, unless otherwise stated, the five corresponding algorithms all run on the same GPU and apply the same experimental configuration settings.

Experiment description

Experiments were conducted on the four benchmark datasets in order to validate the ability of these six models to extract features and their efficiency.

Experiment I. To test the robustness of m-AE. The proposed m-AE has multiple hidden layers, since the number of hidden layers (i.e., the m) significantly affects the precision of feature extraction, the m needs to be firstly verified, i.e., robustness testing of the model, let m set in the range of {1, 2, 3, 4, 5, 7, 10, 15, 20}.

Experiment II. To test the ability of feature extraction for the six models. The six models were run on the four benchmark datasets, and then the testing results were analyzed.

Experiment III. To compare the efficiency of our method with competing methods. These methods were performed on four benchmark datasets and observed their running time.

Ablation experiments. To verify that using the distance metric of rescaling transformation can be beneficial for extracting linearly separable features, the ablation experiments were also designed.

In addition, to eliminate randomness during the experiment and present an objective result, we used cross-validation to verify the six models. We randomly selected two datasets from the four benchmark datasets as the training set to train the six models. Once the six models were well trained, they were tested on the four benchmark datasets, respectively. The process was repeated five times, independently, then we took the average of five testing results was as a measurement.

Experiments on robustness

Results in Fig. 2 show that the performance of the proposed m-AE and the benchmark model AE-BK improves along with increasing of m, and then the performance remains stable when m reaches a certain size, i.e., m = 3. This means that m-AE and AE-BK are not sensitive to large m on the four benchmark datasets, i.e., their network structures are robust within a reasonable range. Therefore, let m be equal to three in subsequent experiments.

Comparisons of accuracy extraction

Results in Table 2 show that the proposed m-AE wins the four competing models and the benchmark model in the accuracy of feature extraction on all considered instances. For competitors, ISSML, ITML and SAE outperform MMD-ISOMAP in most benchmark datasets for the extracted accuracy.

Comparisons of linear separability

The results of ablation experiments in Fig. 3A show that compared with the models without using distance metrics, e.g., AE-BK, SAE, the models using distance metrics (including m-AE, ISSML, ITML) perform much better on most datasets in the extracted accuracy of the features with linear separabilities. Similar, the models using distance metrics also win the model using feature selection, as shown in Fig. 3B. To observe the linear separabilities of the extracted features from the four benchmark datasets, we projected these extracted features onto two-dimensional space, and then visualized them. Figure 4 displays the results of visualized distribution on the four benchmark datasets by the six models. The visualized results show that it is optimal for the separation distance between different types of features extracted by m-AE, meaning that compared with competing and benchmark models, m-AE is a winner in terms of the linear separabilities of the extracted features. Together, these results imply that distance metric-based methods have advantages over feature selection-based methods in terms of extracting the features with linear separabilities.

Running time

Figure 5 displays the running time of methods. Obviously, the advantage of m-AE in running time is not as significant as that in both the extracted accuracy and the linear separabilities of the extracted features. MMD-ISOMAP spends less in running time on most benchmark datasets than distance metric-based methods, meaning that the execution efficiency of feature selection-based methods is higher than that of distance metric-based methods when running upon a high-dimensional space. Distance metric-based methods take a lot of time to calculate the distance between each point pair upon a high-dimensional space, so as to increase the running time.

Insights gained from investigation

Compared with the competitors, the proposed m-AE has outstanding advantage in term of both the accuracy of feature extraction and the linear separabilities of the extracted features on high-dimensional data. We interpret it as following. On one hand, Mahalanobis distance in Eq. (3) can appropriately measure similarities between the input sample and the reconstructed input sample, so as to minimize the loss error of m-AE in Eq. (5). As such, m-AE gains the desired accuracy of feature extraction. On the other hand, we performed a rescaling on K-L divergence metric in Eq. (2) by using A * in Eq. (4), which effectively allows the extracted features to present linear separabilities, because the rescaling can maximized the classification distance between the extracted different types of features. Hence, the features extracted by m-AE present linear separabilities than competitors. Overall, m-AE outperforms the competitors in extracted accuracy and the linear separabilities of the extracted features.

In a high-dimensional space, distance metric-based methods easily evaluates the feature similarity by calculating the distance between the data, however, feature selection-based methods relatively difficulty assess the feature importance. Therefore, distance metric-based methods, e.g., ISSML (Ying, Wen & Shi, 2018) and ITML (Mei, Liu & Karimi, 2014), are more suitable for extracting those low-dimensional features with the linear separability from high-dimensional data than feature selection-based methods. However, the computational time of feature selection-based methods, e.g., MMD-ISOMAP (Bo, Xiang & Zhang, 2016), is lower than that of distance metric-based methods in a high-dimensional space, since distance metric-based methods spend too much in calculating the distance between each point pair.

Although autoencoders have excellent feature capture capabilities, they may perform poorly in extracting linearly separable features, e.g., SAE (Chen, Hu & He, 2018). Whereas, this deficiency of autoencoders can be remedied by introducing a distance metric. Certainly, there are many methods of distance metrics, e.g., Wasserstein distance metric (Lei, Su & Cui, 2019; Zheng et al., 2022), Bhattacharyya distance metric (Mariucci & Reiß, 2017).

Limitations

The ability of the autoencoder to extract linearly separable features depends on the reconstructed data distribution, while the reconstruction of the data distribution is achieved by the Mahalanobis distance metric of rescaling transformation. Upon a high-dimensional space, the calculation of Mahalanobis distance metric is relatively complicated than that up a low-dimensional space. Moreover, matrix factorization operation needs to be implemented for each computation, therefore, the proposed model is trained using large-scale high-dimensional data until it can converge, which may take longer training epoch.