An enhanced CNN-LSTM remaining useful life prediction model for aircraft engine with attention mechanism

Convolutional Neural Network (CNN)

CNN is a special multi-layer perceptron (MLP), which has the traits of local connection and weight sharing. Because of its advantages in image processing, it is widely used in computer vision, image classification and other applications (Zhou, 2020). CNN can be divided into one dimensional, two dimensional and three dimensional CNN (1-D CNN, 2-D CNN and 3-D CNN). We mainly use 2-D CNN, which usually includes convolution layer, activation layer and pooling layer. Figure 1 shows a simple convolutional neural network.

The crucial part of CNN is convolution layer. Convolution layer convolves input data through local connection and weight sharing, and its function is to extract features by shifting convolution filters on original data. Convolution filter is also called receptive field. Convolution filter moves along all dimensions of input data, calculates the weight and the dot product of input, and then adds bias to realize convolution operation. The specific convolution operation is: (1)${\displaystyle {\mathbf{y}}_n={\mathbf{w}}_m\ast {\mathbf{x}}_n+{\mathbf{b}}_m}$

where w_m and b_m ∈ R^h×w represent the weight and bias of the m-th filter, respectively, and their sizes h × w are specified when designing the network, x_n is the n-th region of the input data, and y_n is the convolution output of the filter.

As can be seen from Eq. (1), each convolution filter can generate a corresponding feature map. When multiple filters are specified in a convolution layer, a corresponding number of feature maps can be generated, and the output of the convolution layer can be obtained by stacking all the feature maps in the channel dimension.

The role of activation function is to introduce nonlinear factors to enhance the feature expression ability of the model. At present, the Rectified Linear Unit (ReLU) is the most versatile and effective activation function (Dubey & Chakraborty, 2021), which can improve the network sparsity and reduce the network over-fitting. The expression of ReLU is: (2)${\displaystyle ReLU\left(x\right)=\left\{\begin{array}{cc}\hfill \hfill & \hfill xifx>0\hfill \\ \hfill \hfill & \hfill 0ifx\le 0\hfill \end{array}\right.}$

where x is the input of the activation function.

The role of pooling layer is to gradually shrink the feature space size, decrease the amount of parameters of the network and speed up the calculation. The pooling layer is similar to the convolution layer, and it is also performed by filters. The difference is that the filters of the pooling layer do not perform convolution operation on the input data, but perform pooling operation, which is also called downsamples. Generally, the pooling layer can be divided into average pooling layer and max pooling layer. The average pooling layer is to extract the average value from the filter, and The max pooling layer is to extract the max value from the filter. The existing studies indicate that the max pooling layer is preferable in model training (Raj & Kannan, 2022).

Convolutional block attention module (CBAM)

Attention mechanism is a data processing method in deep learning, which can learn how different parts of the input affect the output (Chaudhari et al., 2021). CBAM is a kind of attention mechanism which could be used in CNN. CBAM is designed as a simple and competent module to be applied to CNN (Woo et al., 2018). Figure 2 shows the structure of CBAM. CBAM is composed of the Channel Attention Module (CAM) and Spatial Attention Module (SAM). CAM and SAM can perform attention operations in the channel and spatial respectively. This structure design enables CBAM to be quickly combined with existing model, saving parameters and computing resources.

As shown in Fig. 3, the CAM firstly performs global average pooling and global max pooling on the input data map D(h × w × c) to output two vectors of size 1 × 1 × c, and inputs them into two-layer MLP respectively which has c/r neurons in the first layer ,where r is reduction rate and c neurons in the second layer. The parameters of these two-layer MLP are shared. Next, add up the outputs of MLP by elements. After that, the result will be activated by sigmoid function to obtain the channel attention A_c. Finally, A_c and D are multiplied by elements to obtain the weighted output D′.D′, and the D′ is input of SAM.

The channel attention mechanism can be expressed by the formula: (3)${\displaystyle {\mathbf{A}}_c\left(\mathbf{D}\right)=\sigma \left(MLP\left({\mathbf{P}}_{\max }\right)+MLP\left({\mathbf{P}}_{avg}\right)\right)=\sigma {\mathbf{W}}_1{\mathbf{W}}_0\left({\mathbf{P}}_{\max }\right)+{\mathbf{W}}_1\left({\mathbf{W}}_0\left({\mathbf{P}}_{avg}\right)\right)}$

where P_max and P_avg are the results of global max pooling and global average pooling of input features, σ is sigmoid activation function, W₀ and W₁ are the weights of the first and second layers of MLP, and their sizes are c/r × c and c × c/r respectively.

As shown in Fig. 4, the SAM uses the data map D′ as input which is the output of CAM, and gets two feature maps with the size of h × w × 1 through max pooling and average pooling operation in channel, and takes these two feature maps as input of convolution layer containing a filter. And the output is activated by sigmoid to get spatial attention A_s. Finally, D′ is multiplied with A_s to get the final feature.

The spatial attention can be expressed as: (4)${\displaystyle A_s\left({\mathbf{D}}^{\prime }\right)=\sigma \left(f_{conv}\left(\left[maxpool\left({\mathbf{D}}^{\prime }\right);avgpool\left({\mathbf{D}}^{\prime }\right)\right]\right)\right)}$

where f_conv is convolution operation, maxpool and avgpool are max pooling and average pooling respectively.

Long and short term memory network (LSTM)

Unlike the general forward feedback network, LSTM is a idiosyncratic kind of recurrent neural network (RNN) (Livieris, Pintelas & Pintelas, 2020). Compared with RNN, LSTM can analyze the input in time series, and it can solve the problem of long-term dependence of the series, avoiding the problems of gradient disappearance and gradient explosion, so it can be used for the analysis of long-term data (Jang et al., 2020).

Figure 5 shows the internal structure of an LSTM unit which can be mainly divided into three parts: forget gate, input gate, output gate. In each unit, the cell state C and the output hidden state h will be updated through the 3 internal gates. The forget gate determines the forgotten message f_t according to the input x_t and the previous hidden state h_t−1, the input gate selects the candidate memory message ${\tilde{\mathbf{C}}}_t$ through the input control i_t to determine the updated content of the previous state C_t−1, and the output gate determines the hidden state h_t that the current cell state passes to the next cell through the output control o_t.

The specific calculation process in LSTM network unit is as follows: (5)${\displaystyle {\mathbf{f}}_t=\sigma \left({\mathbf{f}}_w\cdot \left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{f}}_b\right)}$ (6)${\displaystyle {\mathbf{i}}_t=\sigma \left({\mathbf{i}}_w\cdot \left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{i}}_b\right)}$ (7)${\displaystyle {\tilde{\mathbf{C}}}_t=tanh\left({\mathbf{c}}_w\cdot \left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{c}}_b\right)}$ (8)${\displaystyle {\mathbf{C}}_t={\mathbf{f}}_t\odot {\mathbf{C}}_{t-1}+{\mathbf{i}}_t\odot {\tilde{\mathbf{C}}}_t}$ (9)${\displaystyle {\mathbf{o}}_t=\sigma \left({\mathbf{o}}_w\cdot \left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{o}}_b\right)}$ (10)${\displaystyle {\mathbf{h}}_t={\mathbf{o}}_t\odot tanh\left({\mathbf{C}}_t\right)}$

where f_w, i_w, c_w, o_w are the weights of f_t, i_t, ${\tilde{\mathbf{C}}}_t$, o_t respectively, and f_b, i_b, c_b, o_b are the biases of f_t, i_t, ${\tilde{\mathbf{C}}}_t$, o_t respectively, tanh is tanh activation function.

Dataset description

The dataset used in our experiment is the latest Aircraft Engine Run-to-Failure data set published by NASA Ames Research Center in 2020. The dataset was obtained through C-MPASS platform which could simulate the working process of turbofan aircraft engine, its structure is shown in Fig. 8. Compared with the previous run-to-failure trajectories, it considered the performance degradation behavior of the aircraft engine in real flight conditions, and recorded all the flight environment parameters, operation data and degradation data (Chao et al., 2021).

In order to compare with other methods, we use DS02, the most widely used data set in this data set, which contains the run-to-failure simulation data of nine engines, of which six (unit =2,5,10,16,18,20) are the training set data and three (unit =11, 14, 15) are the test set data. The recorded data of each engine contains four scenario descriptors (W), 13 measured physical properties (X_s), 18 virtual sensors data (X_v) and 10 model health parameters (θ). Table 1 shows the description of monitoring parameters. The units are divided into three flight classes (Flight class 1, Flight class 2, and Flight class 3) according to the length of operation time (short-length flights, medium-length flights, and long-length flights). Figure 9 shows the working conditions of different flight classes of engines in the flight envelope, and the green area in the figure is the flight envelope of the engine.

Figure 10 shows the kernel density estimations of scenario descriptors (W) of each engine. From the distribution, we can see that the working environment of different engines is different. Especially, compared with other engines, the working altitude of unit14 is lower. Under this more realistic condition, the RUL of the aircraft engine is harder to predict.

Experimental process

Data preprocessing

The preprocessing of the monitoring data includes slicing and normalization. Slicing refers to selecting a part of the recorded data of the aircraft engine as the input data of RUL prediction. This is because the monitoring data of aircraft engine in one service cycle contains a large amount of repeated and redundant information. We divide it into 400 copies equally according to the total time in one service cycle, and the data of the corresponding time is extracted and recombined into input data, which can ensure that the input data can describe the RUL information of the aircraft engine, reduce the computing resources and improve the learning speed.

In order to eliminate the training error caused by different measuring units of monitoring data, we normalize the data monitored by different sensors. The normalization method adopted in this article is Min-Max normalization, and the calculation formula is: (11)${\displaystyle \widehat{d}=\frac{d-d_{min}}{d_{max}-d_{min}}}$

d_max, d_min are the maximum and minimum value of the variable, and d, $\widehat{d}$ are the original and normalized values of the variable.

Experimental results and analysis

Comparison and discussion of prediction results

After the training process, input the test set data into the trained model and get the predicted RUL values. Figure 11 shows our RUL prediction results. As shown in Fig. 11, the predicted RUL values of our RUL prediction model are generally distributed near the true values during the whole aircraft engine service life. Therefore, our model can be applied to predict aircraft engine RUL, and the prediction results can provide reliable suggestions for the use and maintenance of aircraft engines.

Table 2:

RUL prediction model parameter setting.

Layer name	Parameters
SequenceInput Layer	size: (400,42,1)
Convolution_2d layer 1	filter size: (10, 3); number of filters: 10; step size: (10, 3)
Maxpooling_2d Layer 1	filter size: (5, 5); padding: same
Convolution_2d layer 2	filter size: (3, 3); number of filters: 10
Maxpooling_2d Layer 2	filter size: (5, 5); padding: same
Convolution_2d layer 3	filter size: (5, 5); number of filters: 10
Maxpooling_2d Layer 3	filter size: (5, 5); padding: same
Channel attention Layer	number of channels: 10; reduction rate: 2
Spatial attention Layer	filter size: (3, 3)
Lstm Layer	number of hidden units: 128
FullyConnected Layer	output size: 1

DOI: 10.7717/peerjcs.1084/table-2

After the RUL prediction result is obtained, the result is evaluated by the evaluation function provided previously to quantify the model performance. In addition, we also use MLP, FNN, CNN and CNN-LSTM (Kong et al., 2019) to predict the test data set. The comparison results are shown in Table 3. Judging from the comparison results in the table, our improved CNN-LSTM based on CBAM prediction model has better performance. Its Score is improved by about 25.9% and its RMSE is reduced by 17.4% compared with other best methods.

Table 3:

Performance comparison with other neural network methods.

Method	RMSE	Score (×105)
MLP	8.34	13.43
FNN	7.89	9.21
CNN	7.14	7.60
CNN-LSTM	6.66	7.80
Ours	5.50	5.78

DOI: 10.7717/peerjcs.1084/table-3

Analysis of prediction process

In the process of RUL prediction, different variables have different influences on the prediction results in the input data map, and this difference can be described by CBAM’s spatial attention in our model. Therefore, we extracted the spatial attention matrix in the 5th, 15th, 25th, 35th, 45th and 55th cycles of unit 11, mapped it into the size of the input data, and the result is displayed in the form of the hot map in Fig. 12.

In the figure, we are able to distinctly see that the weights of different variables are obviously different. A larger weight means that the network will focus more on the changes of the corresponding variables in RUL prediction, while a smaller weight means the opposite. We found four variables with the largest weights, namely HPT_eff_mod, HPT_flow_mod, LPT_eff_mod and LPT_flow_mod, and showed the changes of their values with the service time in Fig. 13. The results show that in the whole life cycle, the changes of the values of most variables are strongly correlated with the service time, so their values also reflect the health of aircraft engines to some extent.