Stock market trading via actor-critic reinforcement learning and adaptable data structure

Research motivation

The complexities of stock market investment have prompted the widespread application of machine learning techniques to enhance profitability and mitigate global capital loss across financial institutions. Traditional investment models predominantly rely on technical or fundamental analysis, employing predefined rules that often lack the flexibility to adapt to dynamic market conditions. In contrast, the proposed RL model utilizes an actor-critic architecture, enabling it to learn directly from interactions with the market environment. This approach facilitates continuous adaptation and decision-making informed by historical patterns and real-time market behavior. The contemporary investment landscape is characterized by significant volatility, frequent disruptions driven by political and economic factors, and heightened competition in global markets. Addressing these challenges necessitates the development of models capable of dynamic adaptation and rapid, accurate decision-making, such as the one proposed in this study. For this reason, we have developed an intelligent model to reduce risk during market trades involving gold, the Euro, oil, and the Dow Jones.

This article proposes a model applying actor-critic RL with an efficient neural network topology. Our proposal determines optimal investment actions in the stock market to maximize profits when the price market is up and avoid capital losses when the market is down. Having analyzed the information in stock time series from different markets, we propose a new temporal data window scheme that identifies the optimal size of the training sale in the actor-critic model. This scheme improves the performance of each agent during the investment and reduces training and processing time. In this way, the obtained results reduce the number of unnecessary investment policies during learning and the losses in investment capital. The intended contributions of this article are:

• 1)

  Creating an adaptable data structure for a specific period of time to enhance the learning and accuracy of investment agents.

• 2)

  Generating specialized investment agents for each stock market and time period to minimize capital loss.

• 3)

  Applying the multi-agent actor-critic RL method to improve investments in various time periods and reduce the model’s learning time.

The article is structured as follows. “Materials and Methods” presents the materials that describe the stock market databases used in this research. Also, this section presents the methods applied as the structure of the reinforcement learning actor-critic model. “Reinforcement Learning Model Proposed” describes our solution approach and the data sets used, detailing the structure of the neural networks in optimizing policies and rewards. “Experiments and Results” shows the RL model’s experiments and results obtained in various markets. In addition, we compare and discuss the results with relevant works and buy and hold strategy to determine the efficiency of the proposed model. Finally, conclusions and future research lines are presented.

Materials

The data utilized in this study were sourced from Yahoo Finance (2024) and encompass historical records for the Crude oil, Gold, and Euro markets, covering the period from 2015 to 2021. These datasets were meticulously preprocessed to ensure completeness, consistency, and suitability for simulations in a realistic experimental environment. The information available includes the date (registration date), open (opening value of market shares), high (highest value), low (lowest value), close (value at closing), and adj close (closing price adjusted for splits and distributions of dividends and/or profit) for each day. Table 3 shows an example of the three stocks (USD).

The crude oil, gold, and Euro markets were selected for their distinct characteristics and their critical roles in global financial systems. Crude oil represents a highly volatile commodity market, frequently shaped by geopolitical events. Gold, in contrast, serves as a traditional safe-haven asset, characterized by relatively stable price trends. The Euro exemplifies the dynamics of a major currency market, reflecting regional economic policies and global financial interactions. Together, these diverse attributes create a comprehensive and challenging testing environment for assessing the adaptability and effectiveness of the proposed reinforcement learning model.

Data preprocessing and visualization data

We used the data markets between 01/01/2015 and 31/12/2021, where each market has 1,772 records and in total 5,316 records. In this dataset, we preprocessed the data by correcting numerous errors with negative values, converting the decimal format from point to comma, and changing the date format from month/day/year to day/month/year. Figure 1 illustrates the performance of the Euro market, which comprises 1,772 records. The minimum opening price is $1.04, while the closing price is $1.13. Additionally, it features maximum and average opening and closing prices of $1.25 and $1.14, respectively. Over the past year, there has been a downward trend attributed to political and social changes worldwide.

Figure 2 illustrates the performance of the gold market, which includes 1,772 records. The minimum opening price is $1,062.00, while the closing price is $1,050.80. Furthermore, it exhibits a maximum opening price of $2,063.00 and a closing price of $2,051.50, with an average opening price of $1,421.36 and a closing price of $1,414.64. Over the past 2 years, price stability has prevailed in the market, largely influenced by gold’s value within the global economy.

Figure 3 illustrates the performance of the crude oil market, which encompasses 1,772 records. The minimum opening price is $13.69, while the closing price is $10.01. Additionally, it features a maximum opening price of $85.41 and a closing price of $84.65, with an average opening price of $54.13 and a closing price of $53.26.

Methods

Actor-critic networks

The core of the RL model is a pair of fully connected (FC) neural networks represented in Figs. 4 and 5.

The actor-network chooses an action $A_t$ at each time step $t$; in turn, the critic network assesses the decision quality based on the input $O_t$. As this network learns which states are better or worse, the actor-network uses this information to teach the agent to look for good states and avoid bad ones. The actor-network learns and updates the weights $w_n$, $w_m$ and $w_p$ of the neural network based on the probabilities $\pi (A_t|O_t;\theta )$, as shown by Soleymani & Paquet (2021).

The actor-network uses 19 inputs (corresponding to the $O_t$ 19 states mentioned above) and three hidden layers for the resulting actions $A_t$. For the hidden layers, we found the best parametric choices for the activation functions and the number of neurons in each layer using a genetic algorithm. The actor-network maps each state $O_t$ generates an output that will be the optimal action $A_t$ to invest in the market. Figure 4 shows the scheme of the actor neural network.

The critic network maps each state $O_t$ (19 inputs $O_t$) to its corresponding action $A_t$ value, assessing its quality. This critic network generates the value $T{D}_{Target}=R_t+\gamma +V({O_t}^{\prime })$ at each learning step, where $T{D}_{Target}$ is the predicted value of all future rewards given the current state $O_t$ and $V({O_t}^{\prime })$ represents the critical-network assessing the next state value ${O_t}^{\prime }$. The advantage function is $T{D}_{Error}=T{D}_{Target}-V(O_t)$, which can be interpreted as a prediction error of the AC agent (Singh et al., 2022).

The advantage function describes whether an $O_t$ state is better or worse than expected. If an action $A_t$ in a state $O_t$ is better than expected (advantage greater than 0), the actor-network is encouraged to keep on performing similar actions $A_t$. However, if it is worse than expected (advantage less than 0), the actor-network is encouraged to adopt other actions. If an action performs exactly as expected (advantage equals 0), the actor-network learns nothing from that action $A_t$.

The critic network (Fig. 5) consists of three FC layers with $Tanh$ and $ReLu$ activation functions. Its output is $T{D}_{Target}$, optimized using the mean squared error loss function. Its weights $w_m,w_n$, and $w_p$ are adjusted (updated) to the new value $T{D}_{Target}$, after each time step $t$. Rectified linear unit (ReLU) neurons were chosen due to their computational efficiency and their effectiveness in mitigating the vanishing gradient problem. This property is particularly critical for optimizing deep neural networks in reinforcement learning applications, where frequent and rapid updates are essential for model performance. To address overfitting, regularization techniques, dropout applied to hidden layers, and temporal cross-validation were employed to enhance the model’s ability to generalize to unseen markets.

The actor-critic method integrates two complementary components: the actor, which determines the optimal action to take in a given state, and the critic, which assesses the quality of these actions by estimating a value function. These components operate in tandem to refine the decision-making policy through the policy gradient and the advantage function, ultimately aiming to maximize long-term cumulative rewards. An actor-critic agent during training estimates probabilities of taking each action $A_t$ in the action space and randomly selects actions based on the probability distribution. The agent interacts with the environment for multiple steps using the current policy before updating the actor and critic properties. An actor-critic agent maintains two function approximators to estimate the policy and value function; first, actor $\pi (A_t|O_t;\theta )$ with parameters $\theta$, outputs the conditional probability of taking each action $A_t$ when in state $O_t$ as a discrete action space where the sum of these probabilities across all actions is 1. Second, critic $V(O,\phi )$, with parameters $\phi$, takes observation $O_t$ and returns the corresponding expectation of the discounted long-term reward. However, during training, the agent adjusts the parameter values in $\theta$. At the end of the training, the parameters remain tuned, and the trained actor function approximator is stored in $\pi (A_t|O_t;\theta )$.

Actor-critic agents use the following training process by applying nine configuration steps. In the first step, we initialize the actor $\pi (A_t|O_t;\theta )$ with random parameter values $\theta$. In the second step, we initialize the critic $V(O,\phi )$ with random parameter values $\phi$. The third step generates N episodes by following the current policy, defined as

(2) $$O_{to},A_{to},R_{(to+1)},...,O_{(to+N-1)},A_{(to+N-1)},R_{(to+N)},O_{(to+N)}$$where $O_t$ is a state observation, $A_t$ is an action taken from that state, $O_t+1$ is the next state, and $Rt+1$ is the reward received for moving from $O_t$ to $O_t+1$. In state $O_t$, the agent computes the probability of taking each action in the action space using $\pi (A_t|O_t;\theta )$ and randomly selects action $A_t$ based on the probability distribution. The starting-time step of the current set of N episodes is $to$, where the beginning of the training episode is $ts=1$ and the terminal state is $O_{(to+N)}$. For each following set of N episodes in the same training episode, $to=to+N$.

In the fourth step for each episode step $t=t_o+1,t_o+2,...,t_o+N$, the return is computed according to

(3) $$G_t=\sum _{k=t}^{t_o+N}({\gamma }^{(k-t)}{R}_k)+b{\gamma }^{(N-t+1)}V(O_{(t_o+N)};\phi ),$$which is the sum of the reward for that step and the discounted future reward. If $O_{(to+N)}$ is not a terminal state, the discounted future reward includes the discounted state value function, computed using the critic network V. However, $b$ is $0$ if $O_{(t_o+N)}$ is a terminal state and $1$ otherwise. The discount factor is defined as $\gamma$.

The fifth step computes the advantage function $D_t=G_t-V(O_t,\phi )$. In the sixth step, it accumulates the gradients for the actor-network by following the policy gradient to maximize the expected discounted reward, as

(4) $$d\theta =\sum _{t=1}^N\mathrm{\nabla }{\theta }_{\mu }\ln \pi (A_t|O_t;\theta )D_t$$

In the seventh step, it accumulates the gradients for the critic network by minimizing the mean squared error loss between the estimated value function $V(O_t,\phi )$ and the computed target return $Gt$ across all N episodes, as

(5) $$d\phi =\sum _{t=1}^N\mathrm{\nabla }\phi (G_t-V(O_t;\phi ){)}^2.$$

In the eighth step, it updates the actor parameters by applying the gradients, defined as $\theta =\theta +\alpha d\theta$, where $\alpha$ is the learning rate of the actor. The ninth step updates the critic parameters by applying the gradients as $\phi =\phi +\beta d\phi$, where $\beta$ is the critical learning rate. Finally, this process is repeated from step 3 to step 9 for each training episode until training is complete, as presented by Mnih et al. (2016).

Experiments settings

For these experiments, we will start with an initial capital of $20,000. Additionally, to identify the best version of the proposed model, we will utilize several dynamic data windows, denoted as $tw$. These windows will include annual ( $an$) data sets (one data set), semiannual ( $sm$) data sets (two data sets with 6 months per year each), and quarterly ( $qu$) data sets (three data sets with 4 months per year each). Referring to the scheme presented in Fig. 6, we have obtained the corresponding data sets for each stock as listed in Table 4.

To carry out the training of the agents, we needed to configure the structure of their neural networks using the parameters in Table 5. The input and output layers have a number of neurons defined by the observations $O_t$, the actions $A_t$, and the update of $T{D}_{Target}$ (described in “Actor-critic networks”) for both actor-critic neural networks. The selection of the number of hidden neurons and layers was the optimal configuration after many experiments. The structure of this neural network (Figs. 4 and 5) contains three hidden layers, with 150 neurons in the Tanh layer and one hidden layer with 100 neurons in the ReLu layer. The Tanh and ReLu activation functions were selected based on earlier studies concerning the multilayer neural networks training for RL (Carapuço, Neves & Horta, 2018; Pham, Luu & Tran, 2021).

The model hyperparameters are described in Table 5, such as the parameter Number of steps to look ahead represents the number of steps the agent interacts with the environment before learning from experience, which was finally fixed at $70$. The Entropy loss weight is a value that promotes agent exploration by applying a penalty (between 0 and 1) for being too sure about which action $A_t$ to take; this variable facilitates avoiding local optima. Gradients are calculated during training, and an extra gradient component is computed to minimize this loss function. Mnih et al. (2016) suggests a value of $0.25$. The Discount factor is applied to future rewards during training, with a value between 0 and 1. In our experiment, we used $0.91$ in Learning rate to define the learning during training: if close to zero, it leads to a very long training time, whereas a value near one may lead to premature convergence; typical values in the literature range between $0.001$ and $0.002$, and we used $0.001$ (Mnih et al., 2016). The Gradient threshold enables networks to be trained quickly, usually not impacting the learned task accuracy (Pascanu, Mikolov & Bengio, 2013); we used a value of $1$. The Maximum number of episodes is the maximum number of training cycles for the agent, after which training terminates. The “maximum number of steps” used was $\text{5,000}$ for the first parameter and $\text{4,000}$ for the second parameter, based on AbdelKawy, Abdelmoez & Shoukry (2021).

As mentioned, in the training phase, data windows of annual, semiannual, and quarterly data were used for each market. Figure 9 presents an example of the progress of the learning curve during training until the trained model obtained 5,000 episodes in the gold market. The yellow line represents the optimal threshold limit of learning, whereas the orange line represents the reward per episode during training. In addition, the blue points are the reward episodes during training, having obtained successful results between 1,500 and 5,000 episodes. This training allowed for generating a model more accurate for a price variation in the stock market through RL.

Evaluation method

To evaluate the performance of an RL actor-critic model applied to neural networks, we used a time series validation approach based on a financial context in which the data exhibited temporal dependence. The historical data of the gold, Euro, oil, and Dow Jones markets were divided into a training dataset containing the first 5 years for each market and a testing dataset with the last 2 years for each market to evaluate the model’s performance. The RL model’s agent was trained to reduce capital loss over time by considering investment actions (sell, buy, and hold). The model observes the accumulated reward at the end of each trading episode. The simulation environment was developed with all financial variables to resemble a realistic trading environment as much as possible. Additionally, a rolling window approach was applied, where the RL model was trained with a dynamic data window to evaluate future data in each iteration.

The model’s performance was validated using a time-series validation approach, where the dataset was divided into training and testing subsets. Furthermore, a rolling window technique was employed to iteratively evaluate the model’s predictions on future data, ensuring robustness and reliability across diverse market scenarios.

Selection method

Fine-tuning of the RL model was conducted through a hyperparameter search using genetic algorithm optimization. This process aimed to refine critical parameters, including the learning rates of the actor and critic networks, the gamma discount factor—which determines the reward time horizon—and the neural network architecture, defined by the number of layers and neurons per layer. These tests were made by applying genetic neural networks to 10% of the data set to determine optimal parameter settings.

The final model was selected based on its performance in time series cross-validation, its optimization of cumulative reward (maximizing profit or minimizing losses), and its stability across different time windows. The model architecture was tested with various network sizes and configurations, ultimately choosing the one that offered the best performance with the lowest reward volatility.

Assessment metrics

In this article, the performance of the agent was assessed using metrics including average profit ( $AverageProfit$), average loss ( $AverageLoss$), and the stability of cumulative rewards across various time periods and data window configurations. We used the following evaluation metrics to measure the performance of the RL actor-critic model in reducing capital loss and managing risk in the gold, euro, oil, and Dow Jones markets:

The average profit, $AverageProfi{t}_t(n)$, refers to the average profit obtained by the agent in the $n$ episodes where it makes successful transactions. This is a fundamental metric in a financial context, as it indicates how much capital the agent generates on average when making the right decisions (buying at a low price and selling at a high price). The $AverageProfi{t}_t(n)$ is relevant because the agent’s main objective is to maximize profits in the long run while trading in volatile environments across different markets. RL seeks to optimize the agent’s policy to increase rewards over time. A high average profit suggests that the agent is learning to identify profitable buy/sell opportunities and is making consistent decisions to maximize the return on each transaction.

The average loss, $AverageLos{s}_t(n)$, measures the average losses incurred by the agent in $n$ episodes where it makes erroneous decisions (buying at high prices and selling at low prices). This is crucial for assessing whether the agent is successfully minimizing losses over time, which is one of the main objectives of this research. The $AverageLos{s}_t(n)$ is a key metric in financial risk management. In this environment, trading errors can lead to significant capital losses, so it is vital that the trader not only maximizes profits but also minimizes the impact of incorrect decisions. A low $AverageLos{s}_t(n)$ indicates that the trader is better at managing risks and avoiding trades that could result in large losses. Minimizing the average loss helps preserve capital and avoid large drawdowns, which is critical to the stability of an investment strategy.

Experiments results

Experimental results, as shown in Figs. 10, 11, and 12A, present simulations conducted over the training (5 years) and testing (2 years) periods. These simulations utilize the RL model and 1-year window data for the Euro, gold, and crude oil stock markets. The figures depict investment actions in the market, including selling stocks (red cross), buying stocks (green star), and holding (no marker) stocks.

The case of the Euro market (Fig. 10) presents the investment simulation, where the model RL carried out 553 sell actions, 697 buy actions, and 573 hold actions. Figure 10 shows the investment price progress from $20,000 to $19,834 and a loss capital of $166.

Figure 11 presents the investment experiment in the gold market using RL. There were 437 sell actions, 437 buy actions, and 886 hold actions in the simulation. The profit from applying RL in the simulation amounted to $1,624.19, increasing from $20,000 to $21,624.19, as depicted.

In contrast, Fig. 12A shows the investment experiment in the crude oil market, where it carried out 313 sell actions, 320 buy actions, and 1,128 hold actions. The profit was $1.06, where the investment price process was from $20,000 to $20,001.06, applying the RL model.

A notable finding was the model’s ability to achieve higher profits in the crude oil market, particularly when utilizing semi-annual data windows. Despite the market’s high volatility, the profits exceeded expectations, highlighting the effectiveness of the adaptive data structure in managing rapid price fluctuations.

The results were obtained for each market over a test period of 2 years with annual, semiannual, and quarterly windows. To assess the proposed policies, the model computes the investment value, using the initial $Initia{l}_{price}$ and current $Curren{t}_{price}$ investment price in equitation 6, where the value is more than 0 $profit=value$ and if $value$ is less than 0 $loss=value$.

(6) $$value=\frac{Curren{t}_{price}-Initia{l}_{price}}{Initia{l}_{price}}$$

The values of profit and loss obtained in this experiment are calculated with $AverageProfi{t}_t(n)$ and $AverageLos{s}_t(n)$. Average profit is the total profit divided by the number of transactions $n$ in $t$ time, as shown in Eq. (7). This value measures how much profit an investor makes on average per trade.

(7) $$AverageProfi{t}_t(n)=\frac{AverageProfi{t}_{t-1}(n)(n-1)+profi{t}_t}{n}$$

On the other hand, the average loss is the total loss divided by the number of transactions $n$ in time $t$, as shown in Eq. (8).

(8) $$AverageLos{s}_t(n)=\frac{AverageProfi{t}_{t-1}(n)(n-1)+los{s}_t}{n}$$

Tables 6 and 7 present the $AverageProfit$, obtained after applying the dynamic data window for each market. For crude oil, we obtained an annual average profit of 4.01%, smaller than the semester of 5.01% (4.78% and 5.24%) and the quadrimester one of 4.48% (4.18%, 4.59% and 4.66%). Similar results were obtained for the gold market, with higher profits in the semester of 4.61% (4.35% and 4.87%) and quarterly 4.24% (4.03%, 4.21% and 4.48%) windows than in the annual one of 4.11%. Finally, for the Euro market, we obtained an annual average profit of 2.37%, a semi-annual profit of 2.40% (2.39% and 2.41%), and a quarterly profit of 2.30% (2.22%, 2.17%, and 2.50%). Thus, in this experiment, the best data window seemed to be the semiannual one, which obtained better profit in every period in the stock market. Globally, the model generated significant profits in approximately 70.83% of tests and losses in only 29.16% of them.

The proposed RL model obtained an $AverageProfit$ of Euro 2.35%, gold 4.32%, and crude oil 4.50% in simulations tests. The obtained results had a Crude Oil profit between 4.01 and 5.01 in tests with an earned capital of $802 (annual), $1,002 (semester), and $896 (quarter). On the other hand, we have obtained the crude oil average loss between 0.01% and 0.21% with $2 (annual), $42 (semester), and $36 (quadrimester). The highest $AverageProfit$ values were obtained with the crude oil and gold market’s 6-month version, where the two first markets received profits, although the third market losses were primarily reduced by more than 1%, as shown in Table 7.

These results were favorable in the simulations compared with related studies and buy-and-hold strategy (B&H) (Atsalakis et al., 2019). In the Euro case, we compared with Chakraborty (2019), with the following results. First, the annualized loss of our RL model was 0.03% vs 9.81%, which decreased the $AverageLoss$ close to zero. In contrast, our model obtained is similar to Chakraborty (2019), with an average profit of 2.37 % vs 2.88% in the annual period. The results obtained through the B&H strategy showed an $AverageProfit$ of 0.007% and an $AverageLoss$ of 0.004% over a span of 2 years as shown in Fig. 13. These findings validate the optimal performance of our RL model.

The reduction in average loss achieved by the model reflects its capacity to minimize financial risks and enhance investment stability. For example, during the test phase, the model successfully reduced losses in the Euro market to 0.03%, demonstrating its effectiveness in managing risk. This metric underscores the model’s ability to mitigate suboptimal decisions and preserve capital, even in volatile market conditions.

The proposed RL model obtained optimal results in the crude oil case, where the average profit was 5.01% and Chakraborty (2019) study was 4.09%. The market behavior maintained a downward trend over long periods of time. In the market, many similar investment operations were made where it was easier to learn for the agent. The results obtained through the B&H strategy (Fig. 14) showed an average profit of 0.074% and an average loss of 0.068% where our RL model obtained higher results.

In the Gold case, we compared the following results with Hirchoua, Ouhbi & Frikh (2021). The annualized average profit of our RL model was 4.11% vs 4.93%. Second, our model in the semester obtained 4.61%. Thus, our RL model had optimal profit in this period due to the adaptable data structure proposed more than Hirchoua, Ouhbi & Frikh (2021). Finally, the results obtained through the B&H strategy in the gold case were higher and showed an average profit of 0.064% and an average loss of 0.068% as shown in Fig. 15.

Transaction costs in the stock market depend on each broker in the U.S., where we have analyzed the fees of four brokers. These fees include stock commission ($0), Section 31 fees (ranging from $0.01 to $0.003 per $1,000), and Trading Activity fees (for selling, buying, and holding stocks, ranging from $0.03 to $0.05 per transaction). On the other hand, investment analysis costs range from $45 to $65 per transaction; however, this expense is not necessary in our proposal. Table 8 presents the fee commissions for each market during the tests of the RL model proposal, with crude oil at $70.88, gold at $88.60, and Euro at $141.76. These values represent less than 1% of the investment capital, making the involvement of an expensive investment analyst unnecessary.

Distinct patterns were identified in the behavior of the three assets. The crude oil market exhibited pronounced volatility and frequent short-term price spikes, necessitating rapid adjustments by the agents. In contrast, the gold market demonstrated relative stability, characterized by gradual trends primarily driven by macroeconomic factors. The Euro market displayed intermediate volatility, reflecting the interplay between regional economic policies and global currency dynamics. These observations underscore the model’s ability to adapt effectively to diverse and dynamic market conditions. The model adapts to sudden market changes through the actor-critic framework, which facilitates real-time policy updates based on observed rewards. The integration of an adaptive data window structure ensures that agents are trained using both historical and recent data, enhancing their ability to respond effectively to abrupt shifts resulting from geopolitical events or economic shocks.

The single-agent model exhibited its poorest performance in the crude oil market, with a significantly lower cumulative gain of $449.57 compared to its performance in the gold and Euro markets. This disparity is evident in Fig. 16, which illustrates a clear divergence in cumulative rewards between the multi-agent and single-agent models in this market.

To validate the effectiveness of the proposed multi-agent model, a comparative analysis was conducted against a single-agent model across three financial markets: Gold, crude oil, and Euro. The results, presented in Table 9, consistently demonstrate the superior performance of the multi-agent approach in terms of final cumulative rewards, absolute differences, and percentage improvements.

The multi-agent model outperformed the single-agent model across all markets. In the gold market, it achieved a 236.53% improvement, underscoring its capability to make efficient decisions in less volatile conditions. Similarly, in the Euro market, the multi-agent approach demonstrated a 276.63% improvement, further highlighting its adaptability and precision in moderately volatile environments.

The crude oil market presented the largest performance gap, with the multi-agent model outperforming the single-agent model by an extraordinary 1,348.64%. The single-agent model achieved only a modest cumulative reward of $449.57, reflecting the challenges posed by the high volatility of this market. In contrast, the multi-agent model demonstrated remarkable adaptability and robustness, achieving a cumulative reward of $6,512.58. These results emphasize the multi-agent system’s ability to manage rapid price fluctuations and navigate complex market dynamics effectively.

The superior performance of the multi-agent model can be attributed to its modular design, which leverages specialized agents trained on specific time windows. This approach enables the model to capture market-specific patterns, reduce training time, and enhance decision-making precision. Consequently, the multi-agent system excels at risk management and mitigates the impact of suboptimal actions, as demonstrated by the substantial performance differences across all markets.

These findings provide compelling empirical evidence supporting the claim that the multi-agent model represents a more effective approach to financial trading, particularly in volatile market conditions. Its dynamic adaptability and ability to optimize investment strategies make it a valuable tool for modern financial systems.

The actor-critic model’s adaptive data windowing enables effective navigation of volatile financial markets, such as crude oil, characterized by rapid price fluctuations due to geopolitical and economic factors. This adaptability allows the model to dynamically adjust its training data windows, identifying short-term patterns crucial in such environments. Consequently, the actor-critic model achieves superior average gains, notably 5.0% in crude oil, while minimizing losses more effectively than traditional methods like DQN, which consistently underperform across various markets (Table 10).

Quantitatively, as Table 10 shows, the actor-critic model demonstrates faster convergence, requiring only 950 to 1,200 episodes across all markets, compared to DQN’s extended convergence time of up to 2,000 episodes. Additionally, it exhibits enhanced stability, indicated by a lower standard deviation in rewards per episode, underscoring its reliability in volatile scenarios where traditional models often falter.

These findings highlight the actor-critic model’s proficiency in adjusting to dynamic market conditions through adaptive data windowing. This leads to improved financial performance and computational efficiency. The model’s robustness and stability make it a valuable tool for automated trading in complex and rapidly changing financial markets.

Limitations/validity

Limitations of this approach, such as reliance on historical data, risks of overfitting, and simplifications in operating costs, are factors that could influence the model’s performance in real market conditions. However, validation methods such as time-splitting and comparisons with traditional strategies provide credibility to the results and demonstrate that the model can be effective in reducing losses in market scenarios with moderately predictable behaviors.

The proposed model offers significant practical applications for financial institutions and individual investors. By minimizing losses and optimizing profits in volatile markets, it can be effectively integrated into portfolio management strategies, serving as an automated tool for real-time investment decision-making. Furthermore, its adaptive capabilities enable it to respond to market fluctuations, making it particularly well-suited for dynamic and unpredictable global financial environments. The model demonstrates robust performance in both bull and bear markets, owing to its dynamic adaptability to changing market conditions. By leveraging short-term patterns through the adaptive data window structure, the model maintains consistent performance, effectively reducing losses during downward trends and capitalizing on upward momentum in bull markets.