2LE-BO-DeepTrade: an integrated deep learning framework for stock price prediction

Introduction

The stock market plays a vital role in the global economy, offering investors the potential for substantial returns while also engendering significant risks. Therefore, accurate prediction of the price movements of individual stocks and the indices used to measure the stock market’s performance is critical for monitoring market volatility and developing investment strategies. However, the stock market, influenced by various dynamic factors such as central bank policy adjustments, changes in monetary policy, geopolitical instability, company policies, investor decisions, and macroeconomic factors, exhibits a complex and unpredictable nature (Rouf et al., 2021). This makes it difficult to predict market movements, according to random walk theory (Fama, 1970).

Stock forecasting is essential for overcoming this unpredictability, which poses significant challenges for investors. Stock forecasting is an interdisciplinary field of study that aims to predict the future price of stock indices or individual stocks. In this field, both traditional and modern methods are employed (Shah, Vaidya & Shah, 2022). While traditional methods rely on the assumption of linear and stationary data structures, modern machine learning and deep learning techniques are better able to model nonlinear relationships and complex data structures (Liu et al., 2022; Shi et al., 2021). In particular, deep learning models have achieved significant success in stock market forecasting due to their multi-layered structures and ability to extract meaningful patterns from large datasets (Akşehir & Kılıç, 2024a; Kanwal et al., 2022; Fisichella & Garolla, 2021; Balasubramanian et al., 2024). However, significant obstacles hinder fully realizing the potential of deep learning (DL) models in financial time series forecasting. A fundamental problem is the inherently high level of noise present in financial data. This noise obscures the underlying patterns, making it difficult for DL models to learn them, thereby degrading prediction performance. A notable gap in the existing literature is the limited systematic integration of advanced signal processing techniques—capable of effectively addressing this noise issue and, particularly, of precisely decomposing different frequency components—into deep learning processes. Another critical problem is the extreme sensitivity of DL models’ performance to architectural choices and hyperparameter settings. This highlights the lack of integrated optimization strategies capable of automatically and efficiently determining the most suitable DL model and its hyperparameters for each signal component, especially when multiple signal components with varying characteristics are obtained after denoising. Finally, a common deficiency is apparent in evaluating the practical value of developed forecasting models; many studies assess model performance solely based on statistical error metrics, yet they often neglect how these metrics reflect the model’s financial viability and profitability in real-world trading scenarios.

The primary motivation of this study is to address these identified problems and gaps in the literature by developing an integrated deep learning framework for stock price forecasting that can produce statistically accurate and practically applicable results. Accordingly, we propose a novel hybrid framework named 2LE-BO-DeepTrade. The proposed framework aims to fill these gaps: (1) It utilizes the 2LE-ICEEMDAN method, which is based on two-level mode decomposition and entropy concepts, to effectively decompose noise in financial time series and obtain cleaner signal components. (2) It applies Bayesian Optimization to automatically optimize the most suitable deep learning model and its hyperparameters for each decomposed IMF component. (3) It evaluates the model’s predictive performance not only through statistical metrics but also in terms of its financial return potential via our newly proposed trading strategy, thereby demonstrating its practical applicability. This integrated approach, by combining noise reduction, model optimization, and comprehensive performance evaluation within a cohesive framework, aims to enhance the accuracy of stock market predictions and the effectiveness of investment strategies.

Related work

Stock forecasting has long been an area of interest for researchers due to the complex structure and dynamic nature of financial markets. Studies in this field primarily encompass data preprocessing, modeling, and evaluation processes. The data preprocessing stage, in particular, stands out as a critical step that directly affects prediction accuracy. Financial data is often incomplete and noisy; therefore, preprocessing the data to remove noise is paramount for improving model performance. The literature includes many studies on preprocessing steps, such as completing missing data, detecting outliers, and noise reduction techniques. In this context, Lv et al. (2022) stated that traditional methods are insufficient for stock index prediction and that noise in financial time series negatively impacts prediction performance. To address this issue, they proposed a hybrid method called CEEMDAN-DAE-LSTM. The method decomposes closing prices into IMFs using Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMDAN), removes redundant data by extracting features with a deep autoencoder (DAE), and uses long short-term memory (LSTM) for prediction. Test results indicate that CEEMDAN-DAE-LSTM provides higher accuracy compared to other methods. In another study, He, Siu & Si (2023) noted that deep learning models are widely used in stock direction prediction, but the limited historical price data for newly listed stocks can reduce performance. To overcome this problem, they proposed an instance-based deep transfer learning model with an attention mechanism (IDTLA). IDTLA enables effective learning with insufficient data and is trained with LSTM model. Similarly, Qi, Ren & Su (2023) developed a hybrid model based on CEEMDAN, wavelet transform, and gated recurrent unit (GRU) for stock index prediction. While CEEMDAN decomposes index data into IMFs, the wavelet transform removes noise in high-frequency IMFs. The GRU model was trained with the denoised data for prediction and tested on the S&P 500 and CSI 300 indices. The results showed that the model outperformed traditional methods such as GRU, LSTM, autoregressive integrated moving average (ARIMA), convolutional neural network-bidirectional long short-term memory (CNN-BiLSTM), and artificial neural network (ANN). Zhao & Yang (2023) noted that machine learning methods are used in stock price direction prediction, but that price data itself makes prediction difficult. To address this issue, they proposed the self-attention based difference long short-term memory (SA-DLSTM) framework, combining sentiment analysis, a denoising autoencoder, and LSTM model. A sentiment index was constructed from forum texts associated with Hang Seng Index (HSI) index price data, features were extracted and noise was reduced using a denoising autoencoder, and direction prediction was performed using LSTM. Compared with traditional methods, the model demonstrated high accuracy and profitability, identifying suitable trading points. Safari & Ghaemi (2025) proposed NeuroFuzzyMan predictor, a novel hybrid model that integrates neuro-fuzzy logic and bidirectional long short-term memory (BiLSTM) networks for financial time series forecasting. This model captures uncertainty in financial data with fuzzy logic and learns long-term dependencies and complex patterns with BiLSTM networks. The model was tested in JPMorgan, Amazon and Tesla stocks, and the results demonstrate the accuracy and robustness of NeuroFuzzyMan using various performance indicators. Cui et al. (2023) stated that stock index prediction is challenging due to the non-linear structure and noise present in stock index data. To overcome this problem, they focused on hybrid models, defining three categories: pure machine learning/deep learning, time series-deep learning, and decomposition-deep learning hybrid models. They proposed the McVCsB model, which combines variational mode decomposition (VMD), convolutional block attention module (CBAM), and BiLSTM methods, for stock index prediction. VMD was used for noise reduction, CBAM for extracting deep features, and BiLSTM for prediction. Experiments on the SSCI, STI, FTSE, and S&P 500 indices showed that the model exhibited superior performance and that its components contributed positively to the prediction. Wang, Liu & Jiang (2024) proposed an interval-valued decomposition integration model for predicting stock indices’ high and low prices, emphasizing that interval-value prediction is more suitable than point-value prediction. The model involves three components: interval multiscale decomposition, multifactor analysis, and nonlinear prediction using intelligent optimization. Stock prices are first decomposed into interval trends and residuals using the interval variational mode decomposition with feedback mechanism (FIVMD). Multifactor analysis examines external factors such as global equity markets and commodity prices, while LightGBM identifies key influencing factors. Dimensionality reduction is done via an autoencoder to reduce complexity. Finally, LSTM, optimized with the whale optimization algorithm (WOA), predicts interval trends and residual series separately. The model was tested on the Shenzhen, Shanghai, and China Securities 100 indices, showing superior performance compared to nine other models. An et al. (2024) proposed the HFSLSMR-LSTM model, combining hierarchical feature selection with local shuffling (HFSLS) and model reweighting (MR) based on LSTM. It was evaluated on stock market indices and individual stock prices using four years of closing price data from YahooFinance for eight indices and 30 DJIA stocks. A total of 158 technical indicators were calculated, and LSTM was used for forecasting after feature selection. The model outperformed six other models, demonstrating its effectiveness in predicting stock prices in volatile markets. Li et al. (2024) proposed the Boruta-MICAN-BSVR method, which combines Boruta feature selection and adaptive noise reduction techniques, for stock index prediction. With this method, important features were selected from 31 technical indicators using Boruta, followed by applying the MIC-ICEEMDAN noise reduction method. After denoising, prediction was performed using support vector regression (SVR) and the brain storm optimization algorithm (BSO). The method was tested on Chinese stock indices (SSEC, SZSE, CSI 500), and compared to eight other methods, it demonstrated lower error rates and higher accuracy. Akşehir & Kılıç (2024c) proposed the 2LE-CEEMDAN method for noise reduction in stock market data, addressing the limitations of existing techniques. A prediction model combining 2LE-CEEMDAN, LSTM, and SVR was used to forecast the next day’s closing prices for indices like S&P 500, DAX, DJI, and SSE. The model outperformed others, proving the effectiveness of this denoising approach. In another study, Akşehir & Kılıç (2024b) introduced the ICE2DE-MDL hybrid model, which applies individualized models to decomposed sub-series, showing superior performance in stock price prediction and highlighting the importance of individualized modeling for noise reduction.

While addressing noise and missing data during the data preprocessing stage is crucial to improve model performance, modeling stock data’s complex and non-linear structure is just as decisive as data quality. However, the complex and non-linear nature of stock data also presents significant challenges in the modeling phase, creating situations that cannot be overcome solely by data preprocessing techniques. Therefore, the modeling methods used to increase prediction success must also be carefully selected. In this context, statistical methods were initially preferred better to model the non-linear structure of stock market data. Although models such as autoregressive moving average (ARMA) (Rather, Agarwal & Sastry, 2015), autoregressive integrated moving average (ARIMA) (Babu & Reddy, 2015), and generalized autoregressive conditional heteroscedastic (GARCH) (Ariyo, Adewumi & Ayo, 2014) have yielded successful results on less complex and low-noise datasets, the complex and non-linear nature of stock market data has caused these methods to be insufficient (Ji, Liew & Yang, 2021; Zhang & Lei, 2022). These limitations have led researchers to more advanced methods, resulting in the proposal of various machine learning and deep learning techniques (Shi et al., 2021). Studies on such innovative methods based on machine learning and deep learning have gradually increased in the literature. For instance, Albahli et al. (2023) proposed a stock trend prediction model to assist investors. The model used historical stock prices and 18 technical indicators, which were reduced in dimensionality using an autoencoder. The reduced features and price data were input into the DenseNet-41 model, which was optimized by reducing the number of Dense blocks from four to three to prevent overfitting. Tested on 10 S&P 500 stocks, the model outperformed other machine learning and deep learning methods, with the DenseNet-41’s feature extraction capability being key to its success. Yao, Zhang & Zhao (2023) introduced the MEMD-TCN hybrid model to address challenges in predicting stock market indexes. The model combines multivariate empirical mode decomposition (MEMD) with temporal convolutional networks (TCN). Stock index data, including various prices and trading volume, were decomposed using MEMD, and the subseries were input into the TCN for prediction. Evaluated on seven stock indexes globally, the model outperformed 14 other models in prediction accuracy, with the combination of MEMD and TCN significantly improving performance. Md et al. (2023) proposed the MLS-LSTM model for stock price prediction, consisting of three sequential LSTM layers and one dense layer. The model used normalized time series data to capture the relationship between past and future prices. Tested on Samsung’s 5-year closing price data, MLS-LSTM outperformed several models, including multi-layer perceptron (MLP), support vector machine (SVM), linear regression, recurrent neural network (RNN), single-layer LSTM, and convolutional neural network (CNN), with an average prediction error of around two percent. Using multiple LSTM layers helped prevent overfitting and enhanced performance compared to single-layer models. Wang & Zhu (2023) proposed a stock trend prediction model integrating sentiment analysis and dual classifier coupling to address the impact of investor-generated textual content on stock behavior. They introduced a sentiment index weighted by reading volume and adjusted for time decay using an exponential moving average. The model combined CNN and SVM for dual classifier coupling. Tested on the SSE 50 and CSI 300 stock indexes, it outperformed models like MLP, SVM, LSTM, and hybrid models (CNN-LSTM, CNN-SGD, CNN-BiLSTM), showing enhanced prediction accuracy with the CNN-SVM model. Yu et al. (2023) introduced the GVMD-Q-DBN-LSTM-GRU hybrid model to predict stock market indices’ realized volatility (RV) with high accuracy. The model employs the gray wolf optimization algorithm and VMD to decompose RV sequences into IMFs. Each IMF is trained using deep belief networks (DBN), LSTM, and GRU, and results are integrated with a Q-learning algorithm to determine optimal weights. Tested on indices like the Shanghai Stock Exchange Composite, FTSE, and S&P 500 using 5-minute data, the model outperformed ten other prediction models based on MAE, MSE, and other performance metrics. Chen, Wu & Wu (2022) reviewed the limitations of deep learning-based models for stock prediction, noting that many studies focus on single-stock data, which can undermine long-term prediction accuracy. The KD-LSTM model was proposed to address this, incorporating a clustering approach using k-means and Dynamic Time Warping (DTW) to group 16 bank stocks with similar price trends on the Chinese stock market. Data from four stocks within the same cluster were used to train the LSTM model. The experimental results showed that this clustering approach enhanced the model’s prediction performance. Yu et al. (2025) proposed a hybrid model (GA-VMD-TCN) based on VMD and TCN, optimized with a genetic algorithm (GA), for multi-step forecasting of stock indices. The model decomposes the stock index into intrinsic mode functions using VMD optimized with GA, and performs multi-step prediction with TCN for each IMF. Experimental results show that GA-VMD-TCN provides better prediction accuracy and stability compared to other methods. Mehtarizadeh et al. (2025) proposed a hybrid model that uses a LSTM network modified with the Sine-Cosine Algorithm (SCA), ARIMA, and GARCH models for stock price prediction. This model aims to manage volatility by effectively capturing linear and non-linear patterns. In the study, experiments with data from 12 different stocks showed that daily predictions outperformed traditional LSTM and other hybrid models by 83.37%, 84.05%, and 55.8%, respectively. Furthermore, the LSTM-SCA-ARIMA-GARCH model demonstrates a significant superiority over other alternative methods.

Considering the methods used in the modeling process and their success, the next critical step is to evaluate the effectiveness and adaptability of these models, thereby measuring prediction accuracy and reliability. Statistical metrics such as MSE, MAE, and R-squared are frequently used to assess prediction accuracy in the literature. Some studies also utilize performance evaluation methods integrated with trading strategies. In this context, Kim et al. (2023) aimed to predict the direction of the KOSPI 200 index while identifying key features used by the model. The study used 30-min KOSPI 200 data and direction labeling based on consecutive data points, considering 16 technical indicators. A CNN model was trained on 16 × 16 image datasets derived from 16-period values. The gradient-weighted class activation mapping (Grad-CAM) method was applied to identify influential features. The results showed that based on Grad-CAM insights, the proposed trading strategy outperformed alternative strategies, including buy-and-hold and those based on common technical indicators. Fang et al. (2023) addressed the limitations of LSTM models in long-term financial time series predictions due to their non-linear and non-stationary nature. They proposed a modified adaptive LSTM model based on an LSTM-BN network with a multi-path parallel structure. The model incorporated batch normalization and dropout layers to prevent overfitting and used a parallel structure with multiple LSTM-BN networks to enhance prediction stability. A voting method determined the final prediction, and an adaptive cross-entropy loss function was introduced for sharp change points in financial data. The model, tested on the S&P 500, CSI 300, and SSE 180 indices, outperformed five other models in terms of accuracy and return. Shah et al. (2022) proposed a hybrid deep learning model combining CNN and LSTM to predict the Nifty 50 stock index’s closing price. The model utilized 48 features, including raw price data, technical indicators, foreign indices, exchange rates, and commodity prices. CNN was used to extract high-level features, while LSTM predicted the next day’s value. The model achieved a mean absolute percentage error (MAPE) of 2.54% on the training dataset, outperforming similar studies. Over 10 years, the CNN-LSTM model yielded a return of 342%, compared to 107% from the buy-and-hold strategy, demonstrating superior performance.

Significant advancements have been recorded in stock forecasting, with numerous studies proposing various methodologies. These studies are summarized in Table 1 concerning their critical approaches, including noise reduction, hyperparameter optimization, the integration of trading strategies, and the utilization of individual modeling strategies. As illustrated in Table 1, while studies such as (Lv et al., 2022) demonstrate the value of employing decomposition techniques for noise reduction, a prevalent limitation in many such works is the omission of systematic hyperparameter optimization for their prediction models or lack of validation through realistic trading strategies. This oversight constitutes a deficiency in pursuing practical and optimal solutions, a core aspect of our problem statement. Similarly, although hyperparameter optimization, as seen in Wang, Liu & Jiang (2024), or trading strategies, as in Zhao & Yang (2023), are encountered in the literature, these approaches often lack holistic integration. The 2LE-BO-DeepTrade framework proposed in this study aims to address these literature gaps. Indeed, as highlighted in Table 1, this framework uniquely offers a comprehensive approach, seldom found in existing literature, by simultaneously integrating (i) advanced and individualized noise reduction with 2LE-ICEEMDAN, (ii) robust hyperparameter optimization, proven effective across multiple deep learning architectures, via Bayesian optimization, and (iii) validation through a concrete profit-loss analysis based on a proposed trading strategy.

Methodology

This section introduces the methodology of 2LE-BO-DeepTrade, the integrated deep learning-based forecasting framework developed for stock price prediction. Data preprocessing techniques, deep learning models, hyperparameter optimization with Bayesian Optimization, commonly used statistical performance metrics, and the PLR-based trading strategy, one of the study’s key contributions, are detailed.

Deep learning models and Bayesian optimization

Within the scope of this study, LSTM, GRU, and LSTM-BN were chosen as deep learning models due to their ability to model complex and long-term dependencies in financial time series. The Bayesian Optimization Algorithm was used for hyperparameter optimization of these models. In this section, all methods used for modeling are explained in detail.

Long short term memory and long short term memory with batch normalization

Recurrent neural networks (RNNs) can process sequential data and model relationships between data points (Elman, 1990). However, RNNs can face challenges such as the “long-term dependency” and “vanishing gradient” problems (Staudemeyer & Morris, 2019; Young et al., 2018). To overcome these issues, LSTM networks, developed by Hochreiter and Schmidhuber, stand out, particularly with their ability to learn long-term dependencies (Hochreiter, 1997).

Unlike traditional RNNs, LSTM has a more complex structure, and thanks to this structure, it has a better capacity for learning and establishing relationships. While the RNN architecture consists of a single layer containing only a tanh layer, the LSTM architecture has a more complex structure consisting of four consecutive layers (Hochreiter, 1997). As seen in Fig. 1, these layers consist of the cell state, output gate, input gate, and forget gate. LSTM enables information to be remembered for long periods and is updated at each time step through the connections between these layers.

Equations (1)–(6) show how the LSTM unit works at time step $t$ for an input vector $X$. In these equations, $i_t$ represents the input gate, $f_t$ the forget gate, $o_t$ the output gate, $C_t$ the memory cell, $h_t$ the hidden state, W the weight matrices, $b$ the bias vector, and $\sigma$ the sigmoid activation function.

(1) $$forge{t}_l=\sigma (W_{forget}\cdot [hidde{n}_{l-1},X_l]+b_{forget})$$

(2) $$inpu{t}_l=\sigma (W_{input}\cdot [hidde{n}_{l-1},X_l]+b_{input})$$

(3) $$outpu{t}_l=\sigma (W_{output}\cdot [hidde{n}_{l-1},X_l]+b_{output})$$

(4) $$\widetilde{Cel{l}_l}=tanh(W_{cell}\cdot [hidde{n}_{l-1},X_l]+b_{cell})$$

(5) $$Cel{l}_l=forge{t}_l\odot Cel{l}_{l-1}+inpu{t}_l\odot \widetilde{Cel{l}_l}$$

(6) $$hidde{n}_l=outpu{t}_l\odot tanh(Cel{l}_l)$$

Fang et al. (2023) proposed the LSTM-BN method for predicting financial time series movements was developed by adding a batch normalization (BN) layer to the traditional LSTM architecture. This additional layer accelerates the training process and enables the model to operate more stably, thereby improving the network’s overall performance.

Gated recurrent unit

GRU, a type of RNN, was developed by Cho et al. (2014) as an alternative to LSTM to address the vanishing gradient problem and capture long-term dependencies in sequential data. As shown in Fig. 2, GRU combines the memory cell in LSTM with the hidden state, reducing the number of parameters and thus offering a more simplified architecture.

GRU fundamentally consists of update ( $updat{e}_l$) and reset ( $rese{t}_l$) gates. These gates control the flow of information within the unit, enabling the hidden state to be updated selectively. The mathematical definition of the GRU cell is given in Eqs. (7)–(10). These equations show how the hidden state is calculated and updated at each time step. Here, $\sigma$ represents the sigmoid activation function, and tanh represents the hyperbolic tangent activation function. The $\odot$ operator denotes element-wise multiplication, W denotes the weight matrix, and $b$ denotes the bias vector.

(7) $$updat{e}_l=\sigma (W_{update}\cdot [hidde{n}_{l-1},X_l]+b_{update})$$

(8) $$rese{t}_l=\sigma (W_{reset}\cdot [hidde{n}_{l-1},X_l]+b_{reset})$$

(9) $$\widetilde{hidde{n}_l}=tanh(W_{hidden}\cdot [rese{t}_l\odot hidde{n}_{l-1},X_l]+b_{hidden})$$

(10) $$hidde{n}_l=(1\text{-}updat{e}_l)\odot hidde{n}_{l-1}+updat{e}_l\odot \widetilde{hidde{n}_l}.$$

Bayesian optimization

Various hyperparameter optimization techniques exist in the literature; commonly used methods such as grid search, random search, and Bayesian optimization stand out. The efficiency of the grid search method decreases as the number of hyperparameters increases. The exponential increase in the number of required evaluations with each additional parameter significantly increases the computational time complexity (Yue et al., 2022). On the other hand, the random search technique uses randomly sampled parameter combinations based on a statistical distribution. However, this approach is insufficient in finding optimal hyperparameter points for some complex models (Yang, Li & Xun, 2019). In contrast to the grid and random search methods, Bayesian optimization determines new parameter values based on past data (Frazier, 2018). This approach provides a more efficient optimization process by making more effective choices, considering the results of previous trials.

Bayesian optimization comprises two fundamental components:

• Surrogate model: This model is used to estimate the probability distribution of the objective function. Based on past evaluation results, this model predicts the possible values of the objective function. Although many surrogate models exist, the Gaussian process is among the most frequently preferred models (Yu & Zhu, 2020).

• Acquisition function: In Bayesian optimization, the acquisition function is used to determine which points in the search space should be evaluated. This function predicts which hyperparameter combinations are likely to be the most optimal, based on past data. The acquisition function serves two primary purposes: first, to provide exploitation to improve the accuracy of the current model, and second, to explore unexplored areas.

The Bayesian optimization algorithm is given in Algorithm 1. This algorithm takes as input the black-box function to be optimized ( $f$), the surrogate model that estimates the objective function ( $\hat{f}$), and the parameter values to be tried (X). At the beginning of the process, a set (H) is created to store the data from previous trials. Using the current surrogate model, the parameter set ( $x_n$) that yields the highest expected output $f(x_n)$ is selected. The actual output ( $y_n=f(x_n)$) is obtained for this selected value. The obtained pairs $x_n$ and $y_n$ are added to the set H to update the past evaluations. This process continues until a specified stopping criterion is met (Cho et al., 2020).

Algorithm 1:

Bayesian optimization.

1: Input: Black-box function f, BO algorithm $\hat{f}$, parameter space X

2: $H\leftarrow \oslash$               ▹ Initialize history set

3: $n\leftarrow 1$                   ▹Initialize iteration counter

4: While true do

5:     $x_n\leftarrow \arg \underset{x\in X}{max}\hat{f}(x;H)$    ▹Find the next parameter set to evaluate

6:     $y_n\leftarrow f(x_n)$  ▹Evaluate the black-box function

7:     $H\leftarrow H\cup \{(x_n,y_n)\}$           ▹Update history

8:    Check stopping criteria

9:     $n\leftarrow n+1$

10: end while

DOI: 10.7717/peerj-cs.3107/table-101

For hyperparameter optimization of the deep learning models in this study, Bayesian optimization utilizing a Gaussian process as a surrogate model was employed. The objective function for this optimization process is defined as a metric that measures the performance of the deep learning model, given a specific hyperparameter set, on the validation dataset. Specifically, this objective function is set to minimize the RMSE value. This choice aims to enhance the model’s generalization ability and prevent overfitting, as the optimization is performed on the validation set rather than directly on the test set. The hyperparameter set yielding the lowest root mean squared error (RMSE) is considered the best model configuration for that particular IMF component.

The 2LE-BO-DeepTrade prediction framework

This study presents an integrated deep learning-based forecasting framework for predicting the next-day closing price of stocks. The proposed model¹ combines the 2LE-ICEEMDAN denoising approach and optimized LSTM, LSTM-BN, and GRU methods. The model’s performance was analyzed not only with statistical evaluation metrics but also with the PLR-based trading strategy developed in this study.

The proposed 2LE-BO-DeepTrade framework for stock price prediction, as schematized in Fig. 3, consists of three main stages:

• 1.

  Denoising with 2LE-ICEEMDAN: The 2LE-ICEEMDAN denoising method is applied to the stock closing price data. The noiseless IMF components obtained in this method’s first and second decomposition stages, determined according to entropy values, are selected for model training.

• 2.

  Construction of the prediction model: Since each IMF component exhibits different dynamic characteristics, a separate modeling approach is adopted for each IMF instead of predicting all components with a single model. Accordingly, using Bayesian optimization, the optimal hyperparameters of LSTM, LSTM-BN, and GRU models are determined for each IMF. Each IMF is predicted separately with the three models trained with these optimized hyperparameters, and the model that provides the lowest MSE value is accepted as the final model for that IMF. The IMFs obtained from the first and second decomposition levels and predicted with the best models are summed to create the final prediction result.

• 3.

  Statistical and financial performance evaluation of the model: The model’s prediction performance is comprehensively evaluated and validated through both statistical metrics and the PLR-based trading strategy developed in this study.

Experimental settings

This study uses Python version 3.10 to implement the classification task for the identification of cerebral infarcts, the detailed configuration equipment is CPU: 12th Gen Intel(R) Core(TM) i9-12900K CPU @ 3.20 GHz, 16 cores, 24 logical processors, 64-bit Windows operating system.

This section first describes the stock datasets used in the study. Then, the search space used in the hyperparameter optimization of the deep learning models is detailed.

Results and discussion

This section evaluated the 2LE-BO-DeepTrade, the proposed deep learning-based integrated stock prediction framework, through experimental studies conducted on different stocks. All stages of the approach developed within the scope of the study were examined and explained in detail on AKBNK, MGROS, KCHOL, THYAO, and ULKER stocks, and the results that were obtained were discussed.

In the first stage, the 2LE-ICEEMDAN denoising method was applied to the closing prices of the five stocks (AKBNK, MGROS, KCHOL, THYAO, and ULKER). For each stock, the entropy values and ratios of the IMFs obtained in the first and second decomposition stages are given in Table 3 ( $Imf-i^{(j)}$ means the i-th IMF as a result of the j-th decomposition). The values shown in bold in the table indicate the IMFs classified as high-frequency, in other words, noisy. The IMF components obtained from the decomposition are visualized in Figs. 4, and 5; here, for example, it is seen that nine IMFs were obtained in the first decomposition stage of the AKBNK stock, and two of them were classified as noisy (Fig. 4A). Similarly, nine IMFs were obtained in the second decomposition stage, and two were identified as noisy. The same analyses were repeated for the other four stocks (MGROS, KCHOL, THYAO, and ULKER), and the number of IMF components and noisy components for each stock were obtained similarly.

The noiseless IMF components obtained from the denoising stage were used in the construction of the prediction models. Accordingly, first, the optimal hyperparameters of LSTM, LSTM-BN, and GRU models were determined for each IMF using Bayesian optimization, and the model that yielded the lowest MSE value among the models trained with these hyperparameters was accepted as the final model for that IMF. The predictions of the IMFs predicted with these best models were summed to create the final prediction result. The hyperparameters obtained from this optimization process performed for the five stocks are presented in Tables 4–8. The final prediction results obtained in this way were used in the next stage to evaluate the model’s performance. In this context, first, the prediction results on the five stocks were evaluated using the statistical metrics presented in Table 9. Additionally, for comparison purposes, the prediction results of the ICE2DE-MDL model, which does not include hyperparameter optimization and was proposed in the study (Akşehir & Kılıç, 2024b), were also added to the same table. When the table is examined, it is observed that the proposed model, although having a simpler structure than the ICE2DE-MDL model, yields more successful results thanks to the best method architectures determined by hyperparameter optimization. This finding demonstrates that model complexity does not always lead to better results; on the contrary, simpler structures can achieve similar or better performance. Furthermore, graphs visualizing the prediction results for each dataset are presented in Figs. 6 and 7 for a detailed examination of the model performance. In the figures, the green curve represents the actual closing values of the stocks, the blue curve represents the data obtained after the denoising process, the red curve represents the predictions of the ICE2DE-MDL model, and the orange curve represents the predictions of our proposed model. When the graphs are examined, it is seen that the predictions of the ICE2DE-MDL model, which used denoised data but did not undergo hyperparameter optimization, have lower accuracy than our proposed model’s predictions. These results demonstrate that supporting our proposed model with hyperparameter optimization improves prediction accuracy by providing a more suitable configuration and parameter selection, thus contributing to obtaining more reliable results.

The model’s performance was also evaluated financially using a PLR-based trading strategy. For this purpose, the strategy, with the number of segments set to 10, generated “Buy” and “Sell” signals for the stocks using the model’s predictions. Assuming an initial capital of 10,000 TL and applying a transaction fee of 0.1% for each trade, trading simulations were performed for the five stocks in the BIST 100 index. The buy and sell signals generated by this PLR-based strategy are shown in Figs. 8 and 9 (green: “Buy”, red: “Sell”), and the profit amounts obtained at the end of an approximately two-and-a-half-year period are shown in Table 10. It was observed that trading based on predictions yielded significantly higher returns compared to actual prices for THYAO, MGROS, and KCHOL stocks, but resulted in lower returns for AKBNK and ULKER stocks. This lower performance, especially for AKBNK, is due to the high volatility of the stock during the period in question. When compared to the “Buy and Hold Strategy” column in Table 10—representing a simple passive investment strategy where the stock is held for the entire period—the PLR-based strategy is seen to provide significantly higher returns for all stocks than this basic strategy. For example, 51 times more profit was obtained for AKBNK, 87 times for MGROS, 19 times for KCHOL, 15 times for THYAO, and 160 times for ULKER. These striking results demonstrate that the PLR-based trading strategy offers significantly higher return potential compared to passive investment strategies and that the model’s predictions can be effectively used to make profitable trading decisions.

Conclusion

In this study, an integrated deep learning-based forecasting framework, named 2LE-BO-DeepTrade, was presented to predict the next-day closing prices of stocks. The framework was evaluated through comprehensive experimental studies on five stocks from different sectors traded on Borsa Istanbul: AKBNK (banking), MGROS (retail), KCHOL (holding), THYAO (aviation), and ULKER (food). The main contributions of this study are: (1) To present a novel forecasting framework that uses the 2LE-ICEEMDAN method for denoising stock prices and combines it with optimized deep learning models. (2) To develop a practical and profitable PLR-based trading strategy based on this framework. The framework’s effectiveness was demonstrated through a multi-stage process involving denoising with the 2LE-ICEEMDAN method, hyperparameter optimization with Bayesian optimization, and performance evaluation using various statistical and financial metrics (trading strategy). The obtained results demonstrate that the proposed model, despite having a simpler structure compared to existing methods like ICE2DE-MDL, achieves superior prediction accuracy thanks to its optimized model architecture and hyperparameters. This finding indicates that complex models may not always yield better results; on the contrary, simpler structures that are appropriately optimized can achieve similar or even better performance. Visual and statistical comparisons between the models support the effectiveness of the proposed approach and show that the optimized hyperparameters significantly improve prediction accuracy and provide more reliable predictions. Furthermore, the PLR-based trading strategy developed in this study has contributed to making more effective investment decisions. Trading operations conducted in accordance with the “Buy” and “Sell” signals generated by the model with the PLR-based trading strategy have yielded higher profits compared to traditional strategies. The proposed strategy provided significantly higher returns for all stocks compared to the passive trading strategy. These results highlight the effectiveness of the proposed PLR-based trading strategy and the potential returns it can provide to investors.

In conclusion, the 2LE-BO-DeepTrade forecasting framework proposed in this study enables accurate and reliable stock price prediction and allows profitable investment decisions to be made using these predictions. This framework, which integrates denoising, optimized deep learning modeling, and financial evaluation, offers a new approach to the field of stock price prediction.

Limitations and future work

While this study demonstrates a promising approach for stock price prediction and trading, it is important to acknowledge certain limitations. The analysis is constrained by the specific timeframe and the limited set of stocks analyzed, and the model’s performance may vary under different market conditions or with different stocks. Furthermore, the PLR-based trading strategy, despite showing significant profitability, also has its limitations. The strategy’s performance, particularly in highly volatile market conditions, was not specifically investigated in this study, and its sensitivity to the number of segments (N), a critical parameter, was not analyzed. Although it is anticipated that the profitability of the PLR strategy might decrease in market conditions involving high volatility, we believe that this profitability can be maximized with an N parameter that is dynamically selected or optimized according to market conditions.

Future research should focus on enhancing the model’s generalizability by expanding the dataset and exploring diverse market conditions. Specifically for the PLR strategy, a comprehensive sensitivity analysis for the ‘N’ parameter should be conducted in line with the consideration above. Methods to optimize this parameter for different market regimes (especially during periods of high volatility) should be investigated, and the strategy’s resilience and effectiveness under various market volatility scenarios should be evaluated accordingly. In addition, exploring alternative hyperparameter optimization techniques, different trading strategies, and incorporating external factors such as macroeconomic indicators or market sentiment could further improve the model’s overall accuracy and robustness. Finally, future work should validate the model’s performance in real-world trading environments and investigate advanced techniques to capture complex market dynamics.

Supplemental Information