A new denoising approach based on mode decomposition applied to the stock market time series: 2LE-CEEMDAN

Related work

Upon reviewing studies focused on noise elimination in financial time series, it becomes evident that denoising approaches relying on auto-encoder, Fourier transform, wavelet transform, and signal decomposition methods are commonly favored. These approaches are chosen to enhance the prediction performance of models susceptible to noise.

The autoencoder-based noise reduction approaches (Zhao & Yang, 2023; Roostaee & Abin, 2023; Rekha & Sabu, 2022) are complex and require intensive computation. Additionally, they lead to the loss of some significant features during the data compression process. This situation negatively impacts the performance of the developed model. Due to all these reasons, this method is not widely preferred for denoising in financial time series.

In the literature, a Fourier transform-based denoising approach has been employed to eliminate noisy components in stock market index data, particularly those that adversely impact the model’s performance in predicting closing prices for the S&P500, KOSPI, and SSE indices (Song, Baek & Kim, 2021). In addition, the Fourier transform has also been used to eliminate the noise in the signals collected from the sensors for the damage detection module (Yang et al., 2021). In these studies, various deep-learning approaches were employed to develop models on denoised data. The experimental results indicated that hybrid models outperformed basic models by combining denoising techniques with deep learning methods. The Fourier transform effectively divides a time series into frequency components and is particularly useful for identifying periodic patterns in the data, aiding in the analysis of continuous-time signals. However, it has limitations in describing time and frequency scale changes in time series data, making it less effective in analyzing time-varying signals (Fourier, 1888). Given its capability to overcome the limitations of the Fourier transform and its significant achievements in signal processing, the wavelet transform is employed in analyzing financial time series (Qiu, Wang & Zhou, 2020).

The wavelet transform-based denoising approach has been applied to eliminate noisy components in different stock market data, and subsequently, LSTM models were developed on the resulting noiseless data (Dastgerdi & Mercorelli, 2022; Tang et al., 2021; Bao, Yue & Rao, 2017). A hybrid noise reduction approach, combining the wavelet transform with complete ensemble empirical mode decomposition with adaptive noise (CEEMDAN) methods, was proposed to enhance further noise reduction in financial time series data (Qi, Ren & Su, 2023). While experimental results in these studies indicate a significant improvement in prediction stability through noise reduction using the wavelet transform, the method has some limitations. These limitations are that the effectiveness of Wavelet Transform in noise reduction depends on the number of decomposition layers and the choice of the basic wavelet function. However, the method’s applicability is restricted, and the effectiveness of noise reduction is limited due to the absence of information about the appropriate values for parameters such as the number of decomposition layers and the choice of the basic wavelet function for time series data (Liu et al., 2022a). Denoising methods based on mode decomposition have gained prominence to address these drawbacks of the wavelet transform.

In the literature, various approaches that decompose time series into different frequency spectra, such as empirical mode decomposition (EMD) (Zhang et al., 2023; Rezaei, Faaljou & Mansourfar, 2021; Lv et al., 2022), ensemble empirical mode decomposition (EEMD) (Wu & Huang, 2009), variational mode decomposition (VMD) (Wang, Cheng & Dong, 2023; Cui et al., 2023), complete ensemble empirical mode decomposition (CEEMD) (Rezaei, Faaljou & Mansourfar, 2021; Yong’an, Yan & Aasma, 2020; Liu et al., 2022b), and CEEMDAN (Cao, Li & Li, 2019; Lv et al., 2022), are frequently preferred for denoising in time series. Examining studies in the literature reveals that the utilization of these approaches, combined with deep learning methods like long short-term memory (LSTM) and convolutional neural network (CNN), can mitigate the limitations of basic/single models (Rezaei, Faaljou & Mansourfar, 2021; Cao, Li & Li, 2019; Lv et al., 2022; Yong’an, Yan & Aasma, 2020; Liu et al., 2022b). Notably, results obtained with hybrid models, incorporating mode decomposition techniques and deep learning methods, demonstrate the superior performance of CEEMDAN-based models over others (Cao, Li & Li, 2019; Lv et al., 2022). When examining the hybrid methods based on mode decomposition applied in the literature to predict stock market closing values, these methods were commonly implemented on indices such as S&P500, HSI, DAX, SSE, and DJIA. Additionally, it is noteworthy that LSTM, a popular deep learning method, is frequently the method of choice in these applications (Rezaei, Faaljou & Mansourfar, 2021; Cao, Li & Li, 2019; Yong’an, Yan & Aasma, 2020). However, in some studies, auto-regressive moving average (ARMA) (Lv et al., 2022) and feed-forward neural networks (FFNN) (Liu et al., 2022b) models are utilized to capture linear components resulting from decomposition. In all these studies, it is seen that the CEEMDAN method is preferred due to its advantages. The advantages of the CEEMDAN method to other mode decomposition methods can be listed as follows (Torres et al., 2011):

• Adaptive noise control: It is a technique that allows the adjustment of the amount of noise within a signal based on the characteristics and variability of the signal. The CEEMDAN method aims to achieve better results in noisy data by using adaptive noise control. In each decomposition stage of the signal, CEEMDAN analyzes the level of the noise component and adjusts the noise amount accordingly. This way, the optimal amount of noise for denoising the signal is determined, and the noise level is adjusted accordingly at each signal stage. As a result, the CEEMDAN method consistently achieves superior results compared to other methods, such as EMD, EEMD, and CEEMD, owing to its effective use of adaptive noise control.

• Improved decomposition: The CEEMDAN method prevents the occurrence of mode mixing, which is a problem in the EMD method where different modes interfere with each other. Mode mixing refers to similar oscillations in different modes or significantly different amplitudes in a single mode. The CEEMDAN method eliminates mode mixing by utilizing adaptive noise control and performing ensemble processing, which involves analyzing the data multiple times. By utilizing these techniques, the CEEMDAN method eliminates mode mixing more effectively than other methods, ensuring a more accurate and reliable signal decomposition.

• Improved signal-to-noise ratio: It provides a better signal-to-noise ratio through adaptive noise control.

This study introduced a two-level denoising approach named 2LE-CEEMDAN, incorporating entropy and CEEMDAN to remove noise from time series data effectively. This novel approach was tested on financial time series data, specifically stock market index data. In the initial step, the denoising technique was applied to the closing prices of stock market indices to eliminate noise from the data. Then, LSTM and SVR methods were applied to the denoised data to estimate the next day’s closing values of the index. This methodology reflects a comprehensive strategy combining denoising and predictive modeling to enhance the accuracy of forecasting financial time series.

Motivation and contributions

In recent years, there has been an increase in research efforts aimed at reducing noise in financial time series data. It has been observed that prediction models created using denoising approaches proposed in these studies achieve more successful forecasting results than basic models. Based on the mode decomposition in the literature, it is evident that removing high-frequency components in methods developing trading strategies results in information loss, adversely affecting the model performances. Addressing how to mitigate this information loss is an open problem.

The primary motivation of this study is to develop a new approach that effectively separates noisy data from financial time series without causing any loss of useful information. Therefore, this aspect distinguishes this study from existing literature, emphasizing a novel perspective.

The fundamental contributions of the study are provided below:

• A novel denoising method, named 2LE-CEEMDAN, has been developed based on CEEMDAN to extract valuable information from high-frequency components discarded as noise.

• A new methodology has been proposed to effectively identify noisy components from the IMFs obtained through the decomposition of time series data by utilizing approximate and sample entropy to measure irregularities in the time series data.

• To demonstrate the method’s effectiveness, a new hybrid prediction model has been presented for predicting the closing prices of stock market indices. This hybrid model includes the 2LE-CEEMDAN denoising approach with LSTM and SVR methods.

Organization

The rest of this study is organized as follows: In the ‘Methodology’ section, the general framework of the proposed prediction model within the scope of the study is given, and the approaches used in this method are detailed. The ‘Experimental Settings’ section mentions the used dataset for the prediction model, how the hyperparameter adjustments of the model are made, and performance evaluation metrics. The results obtained in the ‘Results and Discussion’ section is interpreted by giving them in tables and figures. The results are discussed in the ‘Conclusion and Future Works’ section, and future works are evaluated.

Framework

The problem under investigation in this study is the examination of the impact of noise on a predictive model designed to forecast the price of any commodity. To demonstrate this impact and predict the closing value of the next day’s stock market index, the study proposes a forecasting model called 2LE-CEEMDAN-LSTM-SVR. This comprehensive model has five stages, consisting of two level-based CEEMDAN, entropies, LSTM, and SVR. The framework of the model is visually represented in Fig. 1, with detailed explanations provided below:

• Stage 1-First decomposition: The time series data is decomposed using the CEEMDAN method at this stage. Subsequently, entropy ratios are computed using the approximate and sample entropy values for each IMF obtained from the decomposition process. IMFs with ratios surpassing a predefined threshold value for entropy ratios are classified as high-frequency components, indicating noisy IMF components. Conversely, those with ratios below the threshold are identified as noiseless IMFs.

• Stage 2-Second decomposition: In this stage, which also marks the second step of the 2LE-CEEMDAN method, a different approach is taken compared to directly discarding the IMFs identified as high-frequency in Stage 1. In contrast to directly discarding the IMFs identified as high-frequency in Stage 1, the second decomposition process is applied to these data. This is done because these IMFs may contain valuable information. The procedures outlined in Stage 1 are replicated similarly to categorize the obtained IMFs as either noisy or noiseless. IMFs determined to be noiseless are selected to be used in the training phase.

• Stage 3-Forecasting of second decomposition: The noiseless components identified in Stage 2 undergo separate training using LSTM and SVR methods. The final IMF obtained from the decomposition, often referred to as the residual, is characterized by linear properties and is therefore trained using the SVR model. Conversely, the remaining noiseless IMFs exhibit nonlinear properties and are trained using the LSTM model. The prediction results from each IMF are subsequently combined, providing a comprehensive and integrated forecasting outcome that leverages the strengths of both linear and nonlinear modeling approaches. This multi-model strategy aims to capture and utilize the distinctive features of each component for enhanced predictive accuracy.

• Stage 4-Forecasting of first decomposition: The residue component determined in Stage-1 is trained with the SVR method, and the remaining IMFs are trained with the LSTM method.

• Stage 5-Ensemble unit: It is a simple ensemble unit where the average is calculated after the prediction results from Stages 3 and 4 are summed. Thus, the final prediction result was obtained.

2LE-CEEMDAN denoising method

To comprehend the foundation of the 2LE-CEEMDAN denoising method, it is essential to first detail the explanations of the CEEMDAN and entropy concepts.

Forecasting model

Following the successful removal of noisy components from the time series data through the 2LE-CEEMDAN algorithm, the noiseless IMFs obtained from the first and second decomposition stages constitute the input data for the forecasting model. This process is explained in Stages 3–5, as depicted in the framework presented in Fig. 1. IMFs characterized by nonlinear attributes are fed into the LSTM model. The last IMF (residue) with linear characteristics derived from the decomposition process is also given as an input to the SVR model. After separately were trained each IMF, the prediction results were ensembled. In other words, as shown in Fig. 1, the prediction results in Stage 3 and 4 are combined hierarchically to obtain the final prediction result.

LSTM

Unlike traditional neural networks, recurrent neural networks (RNN) have loops in their architecture and are based on the logic of using sequential information (Elman, 1990). RNN architectures are widely used in many applications, such as translation, voice recognition, language modeling, and handwriting recognition because they can establish and interpret relationships between sequential data (Mikolov et al., 2010). But when implementing these applications with RNN architectures, two main problems arise:

• Long dependency problem: RNN networks can make sense of and interpret data by connecting with the past, thanks to their architecture. But when it goes too far back, it cannot perform this process and establish the necessary connection with the past. In this case, the so-called long dependency problem arises (Olah, 2015).

• Vanishing gradient problem: One of the problems in neural networks that makes it difficult to update the weights of previous layers with backpropagation is the excessive number of layers. The partial derivative is used when calculating the weights of the previous layers in the backpropagation process. Gradient values are calculated with the help of this partial derivative. But when the number of layers increases, the gradient values will approach zero after a certain point in the backpropagation process, and the weights will not be updated over time. In this case, learning will also be difficult for the network. This problem is defined as the vanishing gradient problem (Young et al., 2018).

LSTM networks, a special structure of RNN, were introduced by Hochreiter & Schmidhuber (1997), especially to solve the long dependency problem of RNN. LSTM also solves the vanishing gradient problem, another problem of RNN. Also, LSTM consists of modules that repeat each other, such as RNN. But when the architectures of the two networks are compared, the main difference is the number of layers. The RNN architecture has a single layer containing a tanh layer, while the LSTM architecture in Fig. 2 consists of four consecutive layers, unlike RNN.

LSTM uses these four layers to ensure that information is remembered by the network for long periods. The LSTM transaction equations are given in Eqs. (1)–(6):

(1) $$f_t=\sigma (W_f\cdot [h_{t-1},X_t]+b_f)$$

(2) $$i_t=\sigma (W_i\cdot [h_{t-1},X_t]+b_i)$$

(3) $$o_t=\sigma (W_o\cdot [h_{t-1},X_t]+b_o)$$

(4) $$\widetilde{C_t}=tanh(W_c\cdot [h_{t-1},X_t]+b_c)$$

(5) $$C_t=f_t\odot C_{t-1}+i_t\odot \widetilde{C_t}$$

(6) $$h_t=o_t\odot tanh(C_t)$$

From the above equations, for an input vector X the LSTM unit at time step $t$: it is an input gate, $f_t$ is a forget gate, $o_t$ is an output gate, $C_t$ is a memory cell, $h_t$ is the hidden state, W is weight matrix, $b$ is bias vector and $\sigma$ activation function. The default connections among these units are presented in Fig. 2.

Dataset

To assess the impact of the proposed 2LE-CEEMDAN denoising approach on the prediction model’s performance for one-step-ahead prediction, the following four major global stock indices were selected:

• Standard and Poor’s 500 (S&P500): It is a stock index comprising 500 prominent U.S. companies. It encompasses around 75% of the American stock market.

• Shanghai Stock Exchange Composite (SSE): It is a stock market index that reflects the performance of all A-shares and B-shares listed on the Shanghai Stock Exchange.

• Deutscher Aktien (DAX): It is a German stock market index representing the performance of the 30 largest and most liquid companies trading on the Frankfurt Stock Exchange. These companies are considered major players in the German economy and are carefully selected to provide a comprehensive overview of the country’s stock market.

• Dow Jones Industrial Average (DJI): It is a stock market index that measures the performance of 30 large and well-established companies listed on stock exchanges in the United States. The DJI includes companies from various sectors, such as technology, finance, health care, and manufacturing.

Daily closing values for these stock indices were collected from Yahoo! Finance for the period between January 1, 2010, and October 1, 2019.

The statistical analysis results, including the amount of data, minimum, maximum, average, and standard deviation information for each stock market index, are presented in Table 1. The table shows a substantial range between the minimum and maximum values of the closing prices for the stock market indices, and the standard deviation values are notably high. This indicates that the selected stock market indices exhibit high volatility and possess a non-stationary property.

Construction of the forecasting model

The 2LE-CEEMDAN-LSTM-SVR forecasting model, described in the “Forecasting Model” section, and designed to predict the next day’s closing value of stock market indices, is constructed as follows:

• First, the 2LE-CEEMDAN method, given a pseudo-code in Algorithm 3, was applied to the time series data consisting of closing values for each stock market index. Here, two-level decomposition and then entropy ratio-based denoising was performed using the CEEMDAN method. For the CEEMDAN method used in the decomposition of the stock market index data, the number of trials and white noise standard deviation values were adjusted to 200 and 0.2, respectively, taking reference from the study of Liu et al. (2022b). The values of the embedding dimension and tolerance parameters in approximate entropy and sample entropy calculated for each IMF were determined as 2 and 0.2, respectively, as a result of various experiments. In addition, the noise threshold value for the entropy ratio used to determine noisy and noiseless components was set as 20 (seen Algorithm 3). Therefore, the IMF components with approximate and sample entropy ratios above 20% were noisy, and the rest were noiseless.

• Following the successful elimination of noise from the stock index data, the noiseless IMFs obtained from the first and second decompositions serve as input features for both the LSTM and SVR models. Each IMF component, identified as noiseless through the 1st and 2nd decompositions, undergoes separate training in this phase. The training process involves scaling the IMFs to the range (0,1) using the min-max scaler. Subsequently, the data is divided into training (90%) and test (10%) datasets. While the residual one of these IMFs is trained with SVR, the others are trained with the LSTM model. The final prediction result of the forecasting model is achieved by hierarchically combining the prediction results obtained from the individually trained IMFs, as depicted in Stages 3–5 of Fig. 1.

The hyperparameter settings for the LSTM and SVR prediction models are detailed in Table 2. The hyperparameter tuning process for the SVR method utilized a grid search approach. Hyperparameter values for LSTM were chosen based on relevant literature studies, incorporating commonly preferred values in this study. Experiments were executed by generating training-test sets concerning three different values for the time step parameter specified in the table. Moreover, to mitigate overfitting during training in the LSTM method, a dropout layer with a ratio of 0.1 was introduced between both hidden layers. Additionally, early stopping was implemented.

Performance evaluation

We evaluated the proposed 2LE-CEEMDAN-LSTM-SVR forecasting model using the mean absolute percentage error (MAPE), mean absolute error (MAE), root mean square error (RMSE), and $R^2$, which are often used in the literature. The RMSE, MAE, MAPE, and $R^2$ metrics are calculated using the Eqs. (12)–(15), respectively. In the below equations, $n$ refers to the number of data, P refers to the estimated value, A refers to the actual value, and $\tilde{A}$ refers to the average of actual values.

(12) $$RMSE=\sqrt{\frac{1}{n}\sum _{i=1}^n{(P_i-A_i)}^2}$$

(13) $$MAE=\frac{1}{n}\sum _{i=1}^n|P_i-A_i|$$

(14) $$MAPE=\frac{1}{n}\sum _{i=1}^n|\frac{P_i-A_i}{A_i}|$$

(15) $$R^2=1-\frac{{\sum }_{i=1}^n{(P_i-A_i)}^2}{{\sum }_{i=1}^n{(P_i-\widetilde{A_i})}^2}$$

Comparison with other models

In this study, we conducted an experimental comparison among four models to assess the performance of the recommended 2LE-CEEMDAN-LSTM-SVR forecasting model in the context of stock market prediction. Details regarding these models’ denoising, decomposition, and forecasting methods are presented in Table 5. Among these models, both CEEMD-CNN-LSTM (Rezaei, Faaljou & Mansourfar, 2021) and CEEMDAN-LSTM (Cao, Li & Li, 2019) did not apply any denoising approach to the IMFs; instead, each IMF was individually trained using the specified forecasting model. Additionally, the CAL (Lv et al., 2022) model performed decomposition and denoising processes. This model was applied to the ARMA approach on the IMF, which is more linear as a result of denoising, and the LSTM method was applied to the remaining ones. Finally, the GRU based on CEEMDAN-Wavelet forecasting model (Qi, Ren & Su, 2023) used a combination of decomposition and wavelet transformation for denoising financial time series. The wavelet threshold denoising method was applied to the IMFs obtained from the decomposition, and these IMFs were then reconstructed. Subsequently, the reconstructed IMFs were trained using the GRU method.

The 2LE-CEEMDAN-LSTM-SVR forecasting model proposed in our study uses decomposition and denoising methodologies. Diverging from the approaches mentioned in previous studies, our proposed model employs a two-level decomposition coupled with an entropy ratio-based denoising technique. Moreover, in contrast to the strategy of training all noiseless IMFs with a single forecasting model, as observed in the studies by Rezaei, Faaljou & Mansourfar (2021) and Cao, Li & Li (2019), our model adopts a differentiated approach. Specifically, the more linear components are trained using the SVR method, while the remaining components are trained using the LSTM method. Additionally, the stock market indices utilized by the models in the comparative analysis are detailed in Table 5.

The performance comparison of the 2LE-CEEMDAN-LSTM-SVR prediction model and the baseline methods proposed for intraday stock market prediction is presented in Table 6. Upon examination of the table, our proposed 2LE-CEEMDAN-LSTM-SVR prediction model demonstrates superior performance compared to the CEEMD-CNN-LSTM and CEEMDAN-LSTM models, which lack the denoising approach and solely rely on the decomposition method. Furthermore, in a comparative analysis of experimental results, it is evident that our 2LE-CEEMDAN-LSTM-SVR model exhibits lower error metrics than prediction models utilizing both denoising and decomposition approaches. This outcome validates the effectiveness of our proposed 2LE-CEEMDAN approach in reducing noise in financial time series, demonstrating that the 2LE-CEEMDAN-LSTM-SVR model successfully predicts the closing prices of the next day’s stock market indices.