MFAM-AD: an anomaly detection model for multivariate time series using attention mechanism to fuse multi-scale features

Multivariate time series anomaly detection

In general, the detection of anomalous patterns within data often involves the utilization of methodologies such as the autoregressive integrated moving average (ARIMA) model, exponential smoothing, and moving averages. These methodologies are effective in pinpointing outliers that deviate from the expected trend, leveraging the inherent patterns and statistical properties of time series data.

On the other hand, machine learning approaches leverage algorithms for clustering and classification to detect anomalies. Techniques such as density-based approaches and k-means clustering algorithms are employed to partition high-dimensional time-series data into multiple clusters, facilitating the identification of anomalies that stand out distinctly from other data points. The goal of classification algorithms such as Support Vector Machine (SVM) is to determine the hyperplane with the maximum margin in high-dimensional time series data and use this hyperplane to distinguish between normal and abnormal data points.

However, the escalating complexity of multivariate time series data poses significant challenges for the application of statistical learning and machine learning techniques. As datasets become increasingly intricate, traditional methodologies often encounter limitations in their ability to effectively capture and characterize anomalous behavior.

Deep learning techniques exhibit distinct advantages when addressing the challenge of time series anomaly detection. Models such as long short-term memory (LSTM) and autoencoder (AE) are capable of unraveling complex patterns and intrinsic correlations within data, particularly in scenarios involving high-dimensional or nonlinear data. LSTM can effectively deal with complex multivariate time series data by effectively capturing the inherent long-term dependencies and short-term patterns in sequence data.

For instance, Hawkins et al. (2002) established the fundamental principle of autoencoders within a multilayer perceptron neural network architecture. The method uses reconstruction error as the evaluation index and input variables as the output target to train the model, and realizes the accurate detection of outliers. In contrast, Malhotra et al. (2016) proposed an encoder–decoder model that employed LSTM, demonstrating exceptional effectiveness in processing time-series data generated by sensors monitoring the operational behaviors and health statuses of mechanical devices. This model adeptly pinpoints anomalies in the operation of mechanical equipment by comprehensively analyzing these data streams.

Furthermore, Yang et al. (2023) introduced a dual-attention contrast representation learning-based technique into time series anomaly detection, enhancing the performance of anomaly detection. This innovative method highlights the continuous progress of using deep learning methods to solve the challenge of complex anomaly detection in time series data.

Multivariate time series anomaly detection models are primarily categorized into two general classes: supervised and unsupervised anomaly detection models. Given the scarcity and expense of labeled datasets in real-world settings, unsupervised learning algorithms stand out as particularly advantageous and widely applicable across a diverse range of anomaly detection applications. The challenge of detecting anomalies within multivariate time series data can be efficiently addressed through diverse unsupervised learning strategies, encompassing cutting-edge deep learning techniques as well as conventional classical algorithms. In the absence of prior label information, these techniques adeptly reveal the underlying structure and patterns within the data, enabling precise localization and identification of anomalous patterns. Consequently, unsupervised anomaly detection models such as InterFusion (Li et al., 2021), TranAD (Tuli, Casale & Jennings, 2022), AnomalyBERT (Jeong et al., 2023), and other recent proposals have gained significant popularity in contemporary multivariate time series anomaly detection endeavors.

Multiscale feature time series anomaly detection

Anomaly detection holds significant importance in the field of multi-scale feature time series. The objective is to detect and identify abnormal patterns or behaviors that span multiple time scales. This research domain demands a rigorous examination of time series data, originating from diverse sources like financial markets, biomedical signals, and sensor data. The challenge lies in the simultaneous consideration of multiple time scales, as anomalies can manifest in various forms and scales.

The wavelet transform mode maxima method emerges as a comprehensive approach for anomaly detection across all scales. This method enables the analysis of local signal behavior, effectively pinpointing weak transient and singular features within the time series data. Moreover, mastering the extraction of pertinent information from multi-scale features and its application to anomaly detection is crucial in this field of research.

Furthermore, the utilization of advanced deep learning techniques, such as recurrent neural networks (RNNs) and CNNs, holds the potential to significantly enhance the accuracy and efficiency of multi-scale feature time-series anomaly detection. For instance, Chen et al. (2022) introduced an innovative hierarchical temporal context encoding block that leverages graph convolution and multi-scale extended convolution to effectively detect anomalies in multivariate time series data within the Internet of Things domain. Similarly, Xu et al. (2018) proposed a methodology that utilizes multi-scale time-frequency information extracted from discrete wavelet transforms, in combination with temporal convolutional networks (TCNs), to achieve fault diagnosis in industrial processes.

Recognizing the advantages of deep learning models in unsupervised anomaly detection, we aim to integrate the feature extraction capabilities of CNNs and the temporal dependency modeling abilities of LSTMs to enhance the performance of reconstructing time series data so as to improve anomaly detection.To this end, we introduce a novel time series anomaly detection model named MFAM-AD. This model employs a joint coding approach, incorporating both CNNs and Bi-LSTMs to capture both local and temporal context information from multidimensional time series data across various scales. Additionally, the encoding module incorporates an attention mechanism to effectively realize feature fusion. During the decoding stage, the model leverages FCNNs and Bi-LSTMs for data reconstruction, enabling more precise identification of anomalous behaviors within the time series. Through this innovative approach, MFAM-AD aims to provide a robust and accurate solution for multi-scale feature time-series anomaly detection.

Overview

To effectively detect anomalies, we introduce a novel model, named MFAM-AD, that seamlessly integrates features across multiple time scales. The MFAM-AD model is composed of two fundamental components: the encoder and the decoder. Figure 1 provides a comprehensive overview of the basic architecture of the MFAM-AD model. The encoder plays a crucial role in generating hidden variables by mining intrinsic features such as temporal and multi-scale spatial features of multivariate time series data. This design enhances the model’s detection accuracy, enabling MFAM-AD to effectively consider scenarios where anomalies may occur across various time scales. The decoder uses the hidden variables generated by the encoder to reconstruct the data. These two modules are elaborated upon in detail subsequently.

Encoder

CNNs have advantages in feature extraction, as they can automatically learn the hierarchical representation of input data, thereby eliminating the necessity for manual feature engineering. In this article, we employ a multi-scale, multi-channel CNN to adaptively extract spatial features from time series data. Each channel corresponds to a one-dimensional time series, and the feature extraction process is conducted independently for each channel, yielding local spatial features for each variable. (5)${\displaystyle Y=f\left(W\ast K\ast X+B\right).}$

In this equation, Y represents the feature vector obtained through the convolution operation, where W, K, and B signify the weight, convolution kernel, input data, and bias value, respectively.

The computational efficiency of convolution operations achieved by CNNs through matrix multiplication enables them to automatically learn and extract valuable features from input data, greatly simplifying the feature extraction process. Consequently, CNNs can still maintain a fast training and inference speed when processing large-scale time series datasets. The convolution process applied to the time series is illustrated in Fig. 2. Furthermore, CNNs facilitate automatic feature learning and recognition across multiple scales, eliminating the need for manual design and selection of feature extractors. To capture local spatial features across different scales of time series data, we employ CNNs with varying convolution kernel sizes, enabling adaptive and scale-specific feature extraction.

Three sets of convolutional kernels with dimensions 1∗k_i (i = 1, 2, 3) are used to extract features from time series to extract multi-scale features. These kernels are capable of extracting features from the time series data across different scales, labeled as f₁, f₂, f₃, and spliced together to obtain a feature F with dimensions of 3N (where N is the feature dimension). (6)${\displaystyle F=concat\left(\left[f_1,f_2,f_3\right]\right).}$

The attention mechanism plays a crucial role in emphasizing significant features while suppressing less relevant ones by automatically assigning weights to different features. This capability enhances the model’s effectiveness in utilizing features and focusing on the essential details of the task at hand. Consequently, we fuse multi-scale feature fusion by the attention mechanism as follows:

To compute the attention score, the input matrix X is multiplied by the learnable weight matrix W, which is of the form [3N∗1], resulting in the generation of attention scores (AttnScores). (7)${\displaystyle AttnScores=F\ast W.}$

We then apply the softmax function to the attention scores to obtain the attention weights (AttnWeights). (8)${\displaystyle AttnWeights=softmax\left(AttnScores\right).}$

Finally, to obtain the fused feature (Z), we perform an element-wise weighting operation using the attention weights (AttnWeights) and the original feature (X). (9)${\displaystyle Z=F\ast AttnWeights.}$

After successfully extracting local spatial features from the time series data, the data is forwarded to the Bi-LSTM layer. This enables the model to comprehensively capture the intrinsic temporal dependencies within the data. The Bi-LSTMs are adept at simultaneously encoding sequence information in both forward and backward directions. We denote the forward and backward LSTMs as h_f and h_b, respectively. The final hidden states of the forward and backward LSTMs, denoted by h_f and h_b, respectively, are concatenated to form the spliced hidden variable h. This allows the model to fully capture sequential information in both directions, enabling more accurate temporal pattern recognition. (10)${\displaystyle h=concat\left(\left[h_f,h_b\right]\right).}$

Decoder

LSTMs not only capture the temporal dynamics of data well, but also effectively solve the problems of gradient explosion and vanishing when processing large amounts of sequences. This is enabled by a distinctive internal mechanism referred to as “gates”. Bi-LSTMs, which seamlessly integrate forward and backward LSTMs, provide an enhanced method for incorporating contextual information into output generation, as shown in Fig. 3. By utilizing the design principles of Bi-LSTM, our model is able to comprehensively capture the temporal features of time series data.

In the Bi-LSTM encoder, a sequence of hidden states h = (h₁, h₂, …, h_i) is generated given an input sequence X = (X₁, X₂, …, X_i). Subsequently, the Bi-LSTM decoder utilizes these hidden states h to reconstruct the time series $X^{\prime }=\left(X_1^{\prime },X_2^{\prime },\dots ,X_i^{\prime }\right)$. During the decoding stage, the input data is reconstructed by combining the hidden states in both the forward and reverse orders.

The Bi-LSTM model outperforms traditional RNNs across multiple tasks and offers the aforementioned advantages when processing sequential data. Additionally, it handles noise and missing values in the input data remarkably well. In tasks that involve data reconstruction, the robustness of Bi-LSTMs can mitigate the impact of disturbances such as noise or missing values in the original data on the reconstruction outcomes. Consequently, the decoder component of the MFAM-AD model employs a Bi-LSTM, which is crucial for accurately reconstructing the original data using the hidden variables generated by the encoder. The primary objective of this network is to effectively reconstruct multivariate time series data by thoroughly exploiting the latent temporal features embedded in the hidden variables. By integrating the essential components of LSTMs, Bi-LSTMs emerge as a powerful deep learning model capable of effectively handling complex data with temporal dependencies. The Bi-LSTM decoder in the MFAM-AD model leverages its specialized memory cells and gating mechanisms to precisely capture the hidden temporal features contained within the input hidden variables from the encoder, and uses it to reconstruct the time series.

The decoder is realized through the employment of a Bi-LSTM. Input the hidden variable h containing spatiotemporal features obtained from the encoder into the Bi-LSTM to obtain data X, and output the reconstruction data X’ of the original data through the fully connected network.

The entire process can be formalized using the following equations: (11)${\displaystyle X=\text{BiLSTM}\left(h\right)}$ (12)${\displaystyle X^{\prime }=\text{Linear}\left(X\right).}$

The function BiLSTM represents the operation of the bidirectional long short-term memory network, which processes the hidden variables h to produce the intermediate representation X. In the second equation, the function Linear represents the operation of the fully connected network, which transforms the intermediate representation X into the final reconstructed output X’.

Abnormal scoring and determination

Data reconstruction serves as a crucial component in anomaly detection, aiming to minimize information loss (Xu et al., 2018). The term “reconstruction error” encompasses the potential loss of information incurred during the reconstruction process of input data, forming the fundamental basis of this approach. In this study, the model undergoes initial training using normal data, enabling it to capture the distinctive patterns of normal behavior. Given the relative ease with which normal data can be reconstructed compared to abnormal data, the model demonstrates high accuracy in discriminating between the two. Specifically, the reconstruction error is determined by quantifying the disparity between the input and reconstructed data.

For this purpose, a training dataset S_N is curated from normal time series data, while a test dataset T_A comprises anomalous time series data. The model is trained using S_N, with the mean square loss of the multi-step reconstruction serving as the loss function. The real data at the current time is denoted by x_i, the reconstructed data at the same time is represented as r_i, and the duration of the time window is denoted by n. (13)${\displaystyle Loss=\sqrt{\frac{1}{n}\sum _{i=1}^n{\left(x_i-r_i\right)}^2}.}$

The reconstructed data in the jth dimension at time point i is denoted as $r_i^j$, while the corresponding real data is denoted as $x_i^j$. Then, compute the average of the residuals between the actual data and the reconstructed data across all dimensions to derive the anomaly score. This score serves as the basis for identifying anomalies. (14)${\displaystyle Score=\frac{1}{n}\sum _{j=1}^n|x_i^j-r_i^j|.}$

To determine if x_i is anomaly, we compare its anomaly score with a predefined threshold r. If the score exceeds the threshold, x_i is designated as anomalous; otherwise, it is considered normal. To determine the appropriate threshold r, we utilize the Peaks-Over-Threshold (POT) method, which provides a statistically robust approach for anomaly detection.

Datasets

To validate the effectiveness of MFAM-AD, we employ publicly available datasets, including MSL, SMD1-1, SMAP, NIPS-TS-SWAN, and NIPS-TS-GECCO. Table 1 provides a comprehensive overview of these datasets, outlining their respective characteristics and applications.

The MSL (Mars Science Laboratory) dataset, obtained from the Martian surface, comprises 55 dimensions per node, encompassing crucial parameters such as temperature, humidity, air pressure, and light intensity. Analysis of this dataset enables the detection of anomalies stemming from fluctuations in the Martian environment.

The SMD (Server Machine Dataset) dataset originates from an Internet company, and for our experimental analysis, we have selected the data subset marked as 1-1, referred to as SMD1-1. The SMAP (Soil Moisture Active Passive Satellite) dataset has global soil moisture data, including information on both surface and subsurface soil moisture, measurements of soil moisture in millimeters, percentages of soil moisture profiles, and changes in both surface and subsurface soil moisture.

The NIPS-TS-SWAN dataset (Angryk et al., 2020) serves as a publicly accessible benchmark for multivariate time series analysis, featuring a diverse range of magnetic-field correlation parameters for active regions. Furthermore, the NIPS-TS-GECCO dataset (Lai et al., 2021; Moritz et al., 2018), a drinking water quality dataset for the “Internet of Things”, was introduced at the 2018 Genetic and Evolutionary Computation Conference, offering a comprehensive view of water quality parameters.

Experimental setup

Experimental results

Our MFAM-AD model demonstrates superior performance in comparison to various baseline models, as evident from the comparison experiments outlined in Table 3 and Fig. 4. Across the MSL, SMD1-1, and SMAP datasets, MFAM-AD consistently surpasses all baseline models, except for DCdetector.

Conventional anomaly detection algorithms, such as KNN and PCA, often struggle to effectively capture the diverse and intricate anomalous patterns inherent in time series data. Consequently, their performance often falls short of expectations. Similarly, the AutoEncoder model may not fully capture temporal information during time series reconstruction, thereby limiting its effectiveness and optimization potential. To address these limitations, EncDec-AD incorporates a recurrent neural network into its encoder, enabling precise temporal feature capture. However, this approach may encounter challenges when processing spatial-dimensional features.

The proposed MFAM-AD model employs a CNN network and a Bi-LSTM network to extract both spatial and temporal features, enabling the encoder to accommodate both. By utilizing parallel convolutional kernels of varying sizes, our model can capture intricate patterns in time series data, thereby enhancing anomaly detection capabilities on short sequences. Additionally, through the utilization of Bi-LSTM, the model effectively incorporates both past and future contextual information, thereby more accurately modeling dependencies within sequences. The efficacy of the model is further underscored by its ability to extract both temporal and local spatial features from the time series data, resulting in significantly higher F1 scores on three datasets compared to the baseline model and common machine learning methods. However, our model slightly underperforms compared to the recently proposed DCdetector model. This can be attributed to DCdetector’s ability to directly leverage original data for comparative learning, thus mitigating potential information loss associated with data reconstruction.

We compare our model against the deep learning baseline using various evaluation metrics, including the Matthews correlation coefficient (MCC), affiliation precision/recall pair (Aff-P/Aff-R) (Huet, Navarro & Rossi, 2022), and volume under the surface (VUS) (Paparrizos et al., 2022). The commonly used F1 score fails to capture the impact of anomalous events, whereas Aff-P and Aff-R consider the temporal differences between actual and predicted events. Additionally, the VUS metric incorporates the ROC curve, providing a comprehensive evaluation of the trade-offs associated with abnormal events. These advanced evaluation metrics offer a more thorough and precise perspective on model performance. As shown in Table 4 and Fig. 5, our model achieves significantly higher Aff-P scores compared to the models in the baseline, indicating its superior performance across the three datasets.

To comprehensively evaluate the performance of our model, we conducted comparative experiments with the DCdetector model using the NIPS-TS-SWAN and NIPS-TS-GECCO datasets. The characteristic of these two datasets is that NIPS-TS-SWAN has the highest percentage of anomalies (32.6%), while NIPS-TS-GECCO has the lowest percentage of anomalies (1.05%). As shown in Table 5 and Fig. 6, our model performs more comprehensively on both datasets than DCdetector, suggesting that it is more applicable to difficult anomaly detection tasks.

Ablation studies

The MFAM-AD model encompasses two primary components: the Attention Mechanism Feature Fusion module (AMFF) and the Multi-scale Feature Extraction module (MFE). To assess the individual contributions of each component to the overall model performance, three distinct networks were devised in this ablation study: the Base network, the enhanced Base+MFE network, and the further augmented Base+MFE+AMFF network.

• Base: It represents the baseline model, utilizing a single convolutional kernel for feature extraction.

• Base+MFE: It extends the Base model by incorporating the Multi-scale Feature Extraction module, which employs three distinct convolutional kernels of varying sizes to extract features. The extracted features from different scales are then fused through a straightforward splicing process.

• Base+MFE+AMFF: Building upon the Base+MFE model, this enhanced variant introduces an attention mechanism for feature fusion, thereby augmenting the fusion process with attention-based weighting, further optimizing the integration of multi-scale features.

As shown in Fig. 7, the experimental results suggest that the adoption of a multi-scale feature fusion strategy may inadvertently introduce noise and redundant information, slightly compromising the model’s effectiveness. However, the subsequent integration of the attention mechanism markedly boosted the model’s performance. The combination of the attention mechanism with multi-scale feature fusion effectively mitigates the model’s sensitivity to noise and redundant information, resulting in a substantial improvement in its performance in anomaly detection tasks. The attention mechanism endows the model with the ability to dynamically adjust its focus on features across different scales based on the input data, thereby optimizing its capability to detect anomalies.

Hyperparameter sensitivity

We conducted a series of rigorous experiments to validate several crucial hyperparameters, including the number of encoding layers (It is the hyperparameter n mentioned earlier, hereinafter referred to as Encode-Layer Number) and the number of hidden layer nodes (It is the hyperparameter c mentioned earlier, hereinafter referred to as Hidden-Size).

The number of hidden layer nodes Hidden-Size is an important parameter. As a hyperparameter of the hidden channels, it may have impacts on model performance, memory cost, and running efficiency. We set Hidden-Size ∈{50, 100, 150, 200} as suggested hyperparameters. For all datasets, the performance of the model will improve as the Hidden-Size increases, but in order to consider the inference efficiency of the model, we only set the Hidden-Size to be the largest value among the experimental values.

Many deep models’ performances are dependent on the number of network layers Encode-Layer Number. We set Encode-Layer Number ∈{1, 2, 3, 4} as suggested hyperparameters. Our model can gain the best performance in no more than 3 layers and will not fail with too few encoder layers or overfit with too many encoder layers.

As shown in Fig. 8, the experimental outcomes revealed that these two parameters exert a significant influence on the overall performance of the model. Following tests and a comprehensive analysis of the model’s inference efficiency, it was ultimately determined that an embedding dimension of 200 offers an optimal balance between maximizing inference speed and maintaining model performance. Based on these findings, the optimal number of coding layers was also experimentally confirmed; specifically, the model achieved the best performance when the number of coding layers was set to 1.