Management of investment portfolios employing reinforcement learning

Portfolio management

The theoretical underpinning of portfolio management is extensive, with several key facets deserving recognition. Modern Portfolio Theory (MPT), as introduced by Markowitz (1952), represents a pivotal perspective that has received substantial attention in both academic discourse and among practitioners in the investment arena.

In essence, MPT, as articulated by Markowitz (1952), underscores the opportunity for investors to enhance returns while mitigating risk through strategic diversification. This strategy is rooted in an analysis of historical asset performance and volatility. Subsequently, a mathematical formulation, underpinned by an optimization model, calculates the allocation of each asset within the portfolio. This optimization aims to maximize expected returns for a given level of risk or, conversely, minimize risk. This approach leads to the identification of the most effective portfolio structure likely to yield optimal returns under specific circumstances, often referred to as the minimum variance portfolio (Millea & Edalat, 2022). It is important to note that a higher number of assets in a portfolio doesn’t necessarily equate to prudent diversification. For instance, assets may be concentrated within a single sector, potentially amplifying returns while exposing the portfolio to equivalent risk levels.

Evolutions enveloping the theory emanate across diverse fronts, principally attributable to the challenges associated with complying with the underlying assumptions, which, if unmet, could vitiate any analysis or critique of MPT. Notably, the assumptions of market participant rationality and market efficiency represent characteristics that are incessantly challenged within the literature (Wilford, 2012).

The Efficient Market Hypothesis (EMH) postulates that asset prices in financial markets follow random and independent movements, making it impervious to prediction, even when historical data is analyzed using technical analysis. This is due to the inherent uncertainty of news, which is widely and instantaneously accessible (Fama, 1965). Consequently, the current price is believed to accurately reflect the intrinsic value of the asset, making fundamental analysis unnecessary. The hypothesis asserts that the most reasonable expectation for the future price is the current price, and any return above the market average is considered exceptional.

In contrast, the Adaptive Market Hypothesis (AMH) posits that financial markets undergo cyclical fluctuations between states of efficiency and inefficiency, which are influenced by external factors such as geopolitical conflicts and political interventions (Lo, 2004). This hypothesis further contends that investors display bounded rationality and demonstrate discernible behavioral patterns.

An increase in research efforts directed at formulating investment portfolios has been observed, driven by the proliferation of accessible data and the introduction of innovative methodologies. In light of these advancements, a comprehensive survey by Loke et al. (2023) delineates the developments in the Portfolio Optimization Problem (POP) from 2018 to 2022. The paper categorizes contemporary solution techniques, highlighting key areas, including metaheuristics, mathematical optimization, hybrid approaches, matheuristics, and machine learning.

Significantly, the survey highlights a growing interest in hybrid methodologies, particularly noticeable since 2018. The findings presented by Loke et al. (2023) emphasize the importance of acknowledging and addressing the emerging trends and gaps in this field. This expansion has resulted in noteworthy improvements in the outcomes obtained (Jang & Seong, 2023).

However, there is room for additional exploration within this field, particularly by leveraging artificial intelligence (AI) techniques such as deep learning and reinforcement learning (RL), which have garnered increasing attention in recent research. Furthermore, there are opportunities to integrate non-traditional assets, including cryptocurrencies, commodities, and indices, into such research endeavors (Santos et al., 2022).

Reinforcement learning

Reinforcement learning (RL) represents a subfield within machine learning, with a primary focus on sequential decision-making in uncertain and stochastic environments. The central objective of RL is to determine optimal policies that maximize cumulative rewards over time (Sutton & Barto, 2018). An RL problem comprises fundamental elements, including an agent, environment, states, actions, and rewards. The agent interacts with the environment by executing actions based on its current state, thereby receiving rewards in return, all while endeavoring to acquire a policy that maximizes the accumulated reward.

An essential element in the realm of reinforcement learning is the state-value function, represented as V(s), which quantifies the anticipated value of future rewards accumulated from state s under a specific policy π (Sutton & Barto, 2018). Furthermore, Bellman’s equation, a recursive relationship, establishes a linkage between the current state’s value and the values of subsequent states, thereby facilitating state-value updates and streamlining the pursuit of the optimal policy (Sutton & Barto, 2018). Bellman’s equation for the state-value function is expressed as follows: (1)${\displaystyle V\left(s\right)=\sum _a\pi \left(a|s\right)\sum _{s^{\prime },r}p\left(s^{\prime },r|s,a\right)\left[r+\gamma V\left(s^{\prime }\right)\right],}$ wherein p(s′, r|s, a) denotes the environment transition function, π(a|s) represents the agent’s policy, and γ is the discount factor.

Deep reinforcement learning (DRL) integrates reinforcement learning with deep neural networks, thereby enabling the acquisition of optimal policies in scenarios characterized by high-dimensional state and action spaces (Mnih et al., 2015). DRL has garnered remarkable achievements across a broad spectrum of complex tasks, including gaming, robotics, and system optimization (Silver et al., 2017; Gu et al., 2017; Mnih et al., 2015). This advancement distinguishes itself from conventional RL by its capacity to tackle challenges of greater scale and complexity, which would otherwise encounter computational and representational limitations within the classical RL domain.

The Q-table comprises a matrix that preserves the value associated with each state-action pair, denoted as Q(s, a), where ‘value’ signifies the expected reward upon executing action a in state s and subsequently adhering to policy π (Sutton & Barto, 2018). Conversely, Deep Q-Network (DQN) represents an approach that amalgamates the Q-table concept with deep neural networks, replacing the Q-table with a neural network that approximates the state-action value function, known as the Q-function (Mnih et al., 2015). This modification empowers the algorithm to effectively manage substantially larger and more intricate state and action spaces, generalizing the Q-function while providing superior scalability and efficiency compared to the traditional Q-table. Bellman’s equation for the Q-function is articulated as: (2)${\displaystyle Q\left(s,a\right)=\sum _{s^{\prime },r}p\left(s^{\prime },r|s,a\right)\left[r+\gamma \underset{a^{\prime }}{max}Q\left(s^{\prime },a^{\prime }\right)\right],}$ wherein: Q(s, a) represents the Q-function that depicts the expected value of accumulated future rewards when action a is taken in state s followed by the adherence to the optimal policy. s′ denotes the subsequent state in the environment after executing action a in state s. r indicates the immediate reward procured after performing action a in state s. p(s′, r|s, a) represents the environment’s probability transition function, describing the transition probability to the subsequent state s′ and receiving reward r upon taking action a in state s. γ is the discount factor, ranging between 0 and 1, dictating the relative significance of future rewards compared to immediate rewards. Values close to 0 lead the agent to value immediate rewards predominantly, whilst values approaching 1 make the agent consider future rewards in a more balanced manner. max_a′Q(s′, a′) expresses the maximal value of the Q-function for the subsequent state s′, considering all potential actions a′.

Bellman’s equation for the Q-function facilitates the iterative update of Q-function values, refining the estimates of expected values pertaining to future rewards accumulation, thus contributing to the pursuit of the optimal policy.

Unlike traditional machine learning, which predominantly focuses on supervised or unsupervised learning paradigms, reinforcement learning (RL) techniques are dedicated to sequential learning and decision-making (Goodfellow, Bengio & Courville, 2016). One notable advantage of these methodologies lies in their ability to acquire knowledge directly through interactions with the environment, eliminating the reliance on labeled data. However, it’s important to note that RL techniques can demand substantial computational resources and extended training periods when compared to classical machine learning methods. Additionally, RL and deep reinforcement learning (DRL) problems can exhibit sensitivity to problem formulation, including the specification of rewards and states, necessitating careful adjustments and empirical exploration.

To comprehend the state-of-the-art concerning the application of RL and its variations within portfolio management, Santos et al. (2022) conducted an exhaustive literature review on the seminal works related to the application of artificial intelligence in portfolio management. This systematic literature review renders a comprehensive perspective on the advancements and challenges in the domain, underscoring studies that employ RL and its variants, thereby enabling a more profound understanding of the potential and limitations of these techniques in investment portfolio management.

Literature pertaining to portfolio management via RL

Thanks to technological advancements, the field of finance is subjected to a more comprehensive examination, considering the myriad of available investment strategies. The introduction of RL algorithms has prompted academic literature to reevaluate the domain of investment portfolio management, with a focus on their application and exploration within the financial context. This section highlights some of the key foundational works.

Jang & Seong (2023) employed a deep reinforcement learning technique to enhance the formation of equity portfolios of the S&P 500 index, employing a neural network as the learning agent. The methodology takes into account technical analysis indicators and market trend information to substantiate decisions regarding resource allocation in equities. According to the authors, this approach excels in comparison to other techniques, as it considers asset allocation in accordance with the market trend. The findings suggest that this technique holds promise as an alternative for portfolio optimisation in complex and dynamic financial markets.

In the study steered by Millea & Edalat (2022), DRL found its utility in optimising portfolios that included equities, currency pairs from the Forex market, and cryptocurrencies—a vanguard in relation to prior studies. The authors amalgamated DRL models and hierarchical clustering into a decision-making system to assign weights to an asset portfolio. The DRL agent acquired the proficiency to alternate between low-level models, culminating in superior performance compared to individual models or a random policy. The outcomes indicated that portfolios inclusive of cryptocurrencies exhibited a superior performance in terms of risk-adjusted returns. Nonetheless, it is imperative to accentuate that the cryptocurrency market is highly volatile and harbours significant risks. Thus, the inclusion of these assets in portfolios mandates circumspection and meticulous analysis. Moreover, the authors bolster the notion that ample real-world evidence indicates market efficiency’s shortcomings at various junctures, which implies an ill-suited environment for the application of MPT, consequently substantiating the utility of the adaptations proffered by DRL models.

Song et al. (2022) introduced an innovative method for optimizing investment portfolios utilizing stochastic reinforcement learning. To validate their model, the authors conducted an empirical analysis using data from 22 stocks with the highest trading volume in the S&P 500 index from 2005 to 2020. The model’s performance was evaluated both before and during the COVID-19 crisis. The findings demonstrated that the proposed approach outperformed the benchmark, traditional stochastic models, and popular algorithms, achieving higher returns while maintaining lower risk.

Analysis procedure

For analyzing the results, two subsections were created: one for portfolios that included transaction costs and another for portfolios that did not include these costs.

From this point, a comparison of the accumulated returns of the portfolios with respect to the benchmarks and volatility was initiated. Subsequently, the Sharpe ratio was analyzed, which is based on the returns of the portfolio, benchmark, and volatility as described by Sharpe (1966), thereby capturing the main points for investors in accordance with Markowitz (1952). The Sharpe ratio is calculated as shown in Eq. (9), and a higher value is preferable. (9)${\displaystyle S=\frac{R_p-R_f}{{\sigma }_p}}$

Wherein:

• S denotes the Sharpe Ratio; R_p is the expected return of the investment portfolio; R_f is the risk-free rate; and σ_p denotes the standard deviation of the investment portfolio.

To provide a more comprehensive perspective, the study conducted an analysis of the portfolios’ beta concerning the benchmarks. The beta, as introduced by Markowitz (1952), serves the purpose of assessing whether a specific asset or portfolio exhibits higher or lower volatility compared to the benchmark. The computation of beta, depicted by Eq. (10), indicates that a value below 1 signifies that the portfolio carries less risk than the market, while a value exceeding 1 suggests a higher risk profile. (10)${\displaystyle \beta =\frac{Cov\left(R_i,R_m\right)}{Var\left(R_m\right)}}$

Wherein:

• β_i denotes the beta coefficient of asset i; Cov(r_i, r_m) represents the covariance between the returns of asset i and the market; and Var(r_m) signifies the variance of market returns.

In summary, portfolios with higher returns, lower volatility, a higher Sharpe ratio, and a lower beta are preferable. It is important to note that a Sharpe ratio with a negative value should be interpreted cautiously, as its interpretation is likely misleading due to portfolios with higher volatility being considered superior when analyzing this index in isolation.

In this research, the methodology proposed by Ledoit & Wolf (2008) was employed to compare portfolios with different characteristics. The method uses the bootstrap approach to generate a distribution of the estimated difference in Sharpe ratios and construct a confidence interval to assess its statistical significance. This allows for the detection of significant differences between the compared portfolios.

The specific objective was to analyze the differences between the portfolios, considering transaction costs and the inclusion of commodities in the composition. The analysis sought to understand the impact of these factors on portfolio performance, providing valuable insights for efficient investment management, while benefiting from the robustness of the method used in detecting statistically significant differences between the Sharpe ratios of the compared portfolios.

Portfolios without transaction costs

The performance of portfolios without transaction costs is detailed in Tables 2 and 3. The former exclusively encompasses equities, while the latter includes the incorporation of commodity derivatives. It is important to highlight that the PTF and MINVAR portfolios were constructed using traditional investment methodologies, specifically, Buy-and-Hold and Minimum Variance, respectively. In contrast, the other portfolios leveraged Artificial Intelligence techniques based on reinforcement learning, specifically the five RL algorithms previously mentioned. Notably, the best performance measures for the models are shown in Tables 2–5 by the bold numbers.

When comparing the returns of equity portfolios with the IBOV index, it becomes evident that the conventional MINVAR approach, as well as the SAC and TD3 portfolios, underperformed relative to the benchmark. Conversely, the remaining three AI portfolios outperformed the PTF portfolio in terms of returns. Upon the inclusion of commodities, only the MINVAR portfolio failed to surpass the reference index, whereas all five AI portfolios achieved returns exceeding those of traditional strategies.

Upon aligning the equity portfolio with the DJI, only MINVAR achieved a cumulative return lower than the index. However, the DDPG, PPO, and A2C portfolios once again outperformed traditional techniques. It is worth highlighting that, as indicated by the bold figures in Table 2, the A2C portfolio exhibited remarkable outperformance in terms of cumulative returns. With the inclusion of commodity derivatives in the analysis, only MINVAR yielded results below the benchmark, while the three superior portfolios continued to leverage AI techniques.

Furthermore, one can observe an approximate 12% increase in the average return of the portfolios when commodities are included, an outcome that surpasses both the IBOV and DJI by at least a 10% margin. This emphasises the import of including such assets, in addition to equities, in investment portfolios within volatile environments, corroborating the findings of Millea & Edalat (2022) and indicating a strong return as per Fama (1965). One may view these results in Figs. 1 and 2.

However, an investor should not focus solely on achieving a robust return but should also aim for low volatility to mitigate risk (Markowitz, 1952). The results obtained from the equity investment portfolios, except for the traditional MINVAR technique, exhibited volatility ranging from 19.81% (TD3) to 22.19% (DDPG), a factor potentially associated with the lower returns observed. However, upon the inclusion of commodities, the volatility of all portfolios decreased, emphasizing the significance of incorporating these assets and further supporting the findings of Millea & Edalat (2022). Interestingly, when compared to the benchmarks, none of the equity portfolios displayed higher volatility than the IBOV, and only one portfolio (DDPG) exhibited higher volatility than the DJI. When commodities were taken into account, none of the portfolios exceeded the benchmark in terms of volatility.

When evaluating the Sharpe Index, it becomes evident that the top-performing portfolios consisting solely of equities were A2C, PTF, and PPO, with results differing by less than 0.02 among them. It is apparent that the Buy-and-Hold (B&H) strategy entailed fewer risks than some AI-based portfolios. Conversely, with the addition of commodities to the portfolios, the best-performing choices were SAC, DDPG, and TD3, with differences among them being less than 0.07, showing an increase of approximately 0.2 compared to those containing only equities. Moreover, only the MINVAR portfolio yielded results inferior to the IBOV and DJI, and no portfolio with the addition of commodities underperformed these benchmarks.

In the assessment of the Beta Index, it is evident that none of the portfolios exhibit a higher level of risk compared to the benchmark indices. Notably, beta values decrease with the inclusion of commodity derivatives, reinforcing the consistent pattern of improved results when these assets are integrated. In all cases, the MINVAR portfolio consistently demonstrated the lowest beta value, concurrently exhibiting the weakest performance in terms of returns. This underscores the importance of a more comprehensive analysis, as exemplified by the Sharpe Index. Consequently, it becomes evident that, overall, the portfolios outperformed both the MINVAR and PTF portfolios.

This observation challenges the validity of the efficient market hypothesis, suggesting that the model proposed by Lo (2004) may offer a more suitable framework. The superior performance exhibited by portfolios that integrate technical indicators as influential variables in their evolution lends substantial credence to this perspective.

Portfolios incurring transaction costs

This section addresses the concern regarding the potential costliness of portfolio management when utilizing computational tools, particularly in terms of transaction costs. As demonstrated by the performance indicators outlined in Tables 4 and 5, as well as the visual analysis of portfolio progression depicted in Figs. 3 and 4, it becomes apparent that the inclusion of transaction costs leads to variations in portfolio return outcomes. Notably, these outcomes reveal that returns can even surpass those achieved without transaction costs. This phenomenon arises from the algorithm’s ability to make fewer transactions or opt for different asset acquisitions.

The results presented in Tables 4 and 5 unequivocally indicate that two equity portfolios generated cumulative returns lower than the IBOV (SAC and MINVAR), with only one falling short in comparison to the DJI (MINVAR). However, with the inclusion of commodities in the portfolio, exclusively the MINVAR portfolio exhibited returns below both benchmark indices. In terms of equity portfolios, the top three performers, in sequence, were TD3, PPO, and PTF. With the incorporation of commodities, these rankings shifted to DDPG, TD3, and A2C. Consequently, the returns from AI-driven portfolios once again demonstrate a propensity to outperform those derived from conventional techniques.

Additionally, it is essential to highlight that the mean returns exhibit a notable increase of 9% when commodities are incorporated. In the scenario where only equities are utilized, both mean returns surpass those of the benchmarks. However, upon the inclusion of commodities, the mean returns experience an increase of at least 9% compared to both benchmarks. It is worth emphasizing that the average results remain relatively consistent when comparing portfolios that consider transaction costs with those that do not. The equity portfolio demonstrates a 1% superiority with transaction costs, and a marginally lower performance of nearly 2% when commodities are included.

When examining the volatility of the share portfolios, it is seen that the outcomes are rather similar, with yields approximately 20% per annum, with the exception of the MINVAR portfolio, which falls notably below this average. When commodities are included, the outcomes of all the portfolios also drop. In a manner similar to the portfolios that do not consider transaction costs, none of them achieved a volatility superior to that of the IBOV, nor in relation to the DJI, considering the portfolios with or without commodities.

When analyzing the Sharpe ratio results, it becomes clear that the three top-performing equity portfolios are TD3, PPO, and PTF, in that respective order. However, the differences between them are more pronounced compared to scenarios without transaction costs, with an increase exceeding 0.1. With the inclusion of commodities, we observe that the best-performing portfolios are DDPG, TD3, and A2C, with differences among them amounting to 0.06. While these variations differ in magnitude when compared to scenarios without commodities, there is a consistent overall increase in Sharpe indices, albeit of a lesser scale than in scenarios without transaction costs. Notably, among the equity portfolios, only the MINVAR portfolio yielded returns below those of the IBOV and DJI, and none performed worse when commodities were included.

In the portfolios considering transaction costs, as occurred in those that did not consider them, it is observed that the β index is less than 1 for all cases, and the inclusion of commodities results in a reduction of all their outcomes.

Lastly, it is worth emphasizing that, as with the portfolios that did not take into account transaction costs, the portfolios, in general, presented superior results to MINVAR, which is based on the minimum-variance technique, and some outcomes were superior to PTF, which is based on the B&H strategy. It is vital to consider the relevance of these outcomes for the development of more effective investment strategies.

Statistical tests

To ascertain the statistical superiority of portfolio performances, we employed the Ledoit-Wolf Test (LW Test) developed by Ledoit & Wolf (2008) to analyze variations in Sharpe Ratios. Our analysis consisted of several comparisons to shed light on the performance of various strategies under different portfolio scenarios. The outcomes are summarized in Tables 6 and 7.

Regardless of the particular case, our investigation showed that the MINVAR method consistently underperformed the other techniques. In the portfolio that included commodities but excluded transaction expenses, TD3 showed a statistically significant lower Sharpe ratio than A2C, DDPG, and SAC. However, despite the fact that A2C appeared to perform better than the others, there was no discernible difference between DDPG, SAC, or A2C.

To further scrutinize the disparities in Sharpe ratios among all models in different investment scenarios, we conducted the LW Test across three key comparisons: “Commodities” evaluates the impact of including this asset class in portfolios; “Transaction Costs” which examines the effects of taking into account transaction costs; and “Both”, that combines both factors. These results are detailed in Table 8.

According to our investigation, all models and the combined scenario showed statistically considerably better results for portfolios that included commodities. The impact of transaction costs on portfolio performance, however, did not show any statistically significant changes.

The robustness of these results suggests that certain models excel in specific contexts, offering useful evidences for both investors and academics. Understanding which models perform optimally when considering factors as the inclusion of commodities and transaction costs can guide more effective portfolio management. Thus, these findings can be deemed not only interesting but also relevant to the fields of investment and finance.

Portfolio composition analysis

The examination into asset weight distribution, classified by countries or commodities, uncovers intriguing variations across the models and portfolio types under scrutiny. Detailed outputs can be found in Table 9.

Specifically, when examining portfolios that exclude both transaction costs and commodities (Table 9), significant disparities in investment allocations to China and Europe emerge among different models. Notably, the TD3 model stands out with a preference for China, reflecting a weight of 0.32. In portfolios that incorporate commodities but do not account for transaction costs, all models–except SAC–display a nearly uniform distribution among asset classes. However, when transaction costs are included, TD3 assumes a more substantial position in Brazil and the USA.

In portfolios that comprise both stocks and commodities without considering transaction costs, the TD3 model exhibits a notable inclination towards commodities. This preference is evident in its highest maximum, minimum, and average weights (25%, 20%, and 23.4%, respectively) and low standard deviation. In contrast, the other models maintain a more balanced allocation among asset classes.

Regarding portfolios exclusively composed of stocks, most models maintain nearly equal distribution among various countries. However, SAC and TD3 display a slight preference for the European and Chinese markets, respectively. These variations in weight allocations signify the unique strategies and learning patterns employed by each model, often shaped by diverse hyperparameter configurations. This highlights the significance of considering multiple approaches when constructing diversified investment portfolios.

Moreover, Table 9 provides extra and intriguing data that offers implicit deductions about these findings. As supplementary information, the complete historical data, including weight distribution, is accessible in the data repository, providing a comprehensive resource for further analysis.