Prediction model of stock return on investment based on hybrid DNN and TabNet model

Xgboost algorithm

Based on the boosting framework, the XGBoost algorithm can be regarded as an improved version of the GBDT model. This algorithm takes the decision tree as the base function and combines the additive model (the combination of weak learners) with the forward distribution algorithm to realize the optimization process of learning. Specifically, the basic idea of the XGBoost algorithm is to continuously split the input data to grow a CART tree, where each tree corresponds to a new function to fit the residuals of the previous tree predictions, namely the gradient boosting method (Friedman, 2001). Ultimately, the characteristics of the sample correspond to a leaf node in each tree with a predicted value. The predicted result for the sample is obtained by summing these values.

It can be seen from the above that making the value of the objective function (loss function) as close to zero as possible is the core objective of the XGBoost algorithm. The XGBoost algorithm improves upon the loss function from the gradient boosting algorithm, becoming a more efficient boosting algorithm. The loss function (Chen & Guestrin, 2016) can be written as: (1)${\displaystyle Obj=\sum _{i=1}^{m}l\left(y_i^{{}^{\prime }},y_i\right)+\sum _{k=1}^k\Omega \left(f_k\right),}$ where i represents the ith sample in the data set; m is the total amount of data in the kth tree; k stands for the kth CART tree generated in the model. ${\mathrm{y}}_{\mathrm{i}}^{{}^{\prime }}$ and y_irepresents the predicted value and the real value respectively. l is the empirical loss function of based-tree model, which is mainly divided into square loss function and logistic regression loss function. Different from GBDT algorithm, the regular term Ω is added to the loss function of XGBoost model to control the complexity of the model and reduce the possibility of over fitting, where Ω can be written as: (2)${\displaystyle \Omega =\gamma T+\frac{1}{2}\lambda {∥w∥}^2,}$ where T represents the number of nodes in the CART tree, $\frac{1}{2}{∥w∥}^2$ means L2 regularization of the value w of the node. γ and λ are super parameters, which are used to control the weight of T and w. The mapping relationship q between the weight vector w and the leaf node is as follows: (3)${\displaystyle f_t\left(\mathbf{x}\right)=w_{q\left(x\right)},w\in {\mathit{R}}^{\mathit{T}},q:{\mathit{R}}^{\mathit{D}}\to \left\{1,2,\dots ,T\right\},}$

Because XGBoost model is essentially an additive model, its final prediction score is the cumulative sum of the scores of each weak learner. Assuming that the CART tree generated in iteration t is ft, there is: (4)${\displaystyle y_i^{{}^{\prime }\left(t\right)}=\sum _{k=1}^t{f}_k\left({\mathbf{x}}_i\right)=y_i^{{}^{\prime }\left(t-1\right)}+f_t\left({\mathbf{x}}_i\right),}$

Bring Eq. (4) into Eq. (1), and the loss function can be expressed as: (5)${\displaystyle Ob{j}^{\left(t\right)}=\sum _{i=1}^{n}l\left(y_i,y_i^{{}^{\prime }\left(t\right)}\right)+\sum _{k=1}^k\Omega \left(f_k\right)=\sum _{i=1}^{n}l\left[y_i,y_i^{{}^{\prime }\left(t-1\right)}+f_t\left({\mathbf{x}}_i\right)\right]+\Omega \left(f_t\right),}$

Then, the loss function is approximated to obtain the optimal solution. The traditional GBDT algorithm only uses the first derivative information of the objective function, resulting in greater information loss. The improved XGBoost algorithm performs a Taylor expansion on the objective function, while retaining both the first-order and second-order derivative information. This approach is conducive to more accurate and faster gradient descent. Therefore, the approximate loss function can be expressed as follows: (6)${\displaystyle Ob{j}^{\left(t\right)}\approx \sum _{i=1}^n\left[g_i{f}_t\left({\mathbf{x}}_i\right)+\frac{1}{2}{h}_i{f}_t^2\left({\mathbf{x}}_i\right)\right]+\Omega \left(f_t\right),}$ where g_i and h_i are the first-order and second-order partial derivatives of the loss function l, respectively. Divide all samples xi belonging to the jth leaf node into a leaf node sample set, that is (7)${\displaystyle I_i=\left\{i|q\left({\mathbf{x}}_i\right)=j\right\},}$

Then combine Eq. (2), Eq. (3) and Eq. (6), we can get:

${\displaystyle Ob{j}^{\left(t\right)}\approx \sum _{i=1}^n\left[g_i{f}_t\left(x_i\right)+\frac{1}{2}{h}_i{f}_t^2\left(x_i\right)\right]+\frac{1}{2}\lambda {\left|w\right|}^2}$ (8)${\displaystyle =\sum _{j=1}^T\left[\left(\sum _{i\in I_i}{g}_i\right)w_j+\frac{1}{2}\left(\sum _{i\in I_i}{h}_i+\lambda \right)w_j^2\right]+\gamma T,}$

It can be seen that there is only one variable in Eq. (8), namely, the weight vector w. This is because the objective function of each leaf node is completely independent. That is, when the sub-objective functions of each node obtain their optimal values, the final objective function, Obj, can also achieve its optimal value. Once the structure of the model has been determined, the optimal target values of Obj and the leaf node weights w_j can be obtained by deriving Eq. (8), as shown in Eqs. (9) and (10): w_j

(9)${\displaystyle w_j^{\ast }=-\frac{G_j}{H_j+\lambda },}$ (10)${\displaystyle Ob{j}^{\left(t\right)}=-\frac{1}{2}\sum _{j=1}^T\frac{G_j^2}{H_j+\lambda }+\gamma T}$ where G_j stands for ∑_i∈Iig_i, H_j represents ∑_i∈Iih_i + λ, both are constants. In addition, the determination of the model structure is consistent with the construction of the decision tree. The greedy algorithm is used to select the sub-CART structure that can minimize the system’s entropy; that is, each node segmentation is based on making the increment of the Obj value as large as possible.

Beyond the modifications and optimizations of the GBDT algorithm, the XGBoost algorithm has also undergone significant optimization in engineering aspects, such as supporting parallel operations and effectively using hardware resources. These improvements make it superior to its predecessors in terms of accuracy and operational speed. However, the algorithm still has some shortcomings, such as having too many parameters that are difficult to adjust, being only suitable for dealing with structured data, resulting in a heavy dependency on feature engineering, and an inability to handle high-dimensional feature data, among others.

LightGBM algorithm

To address the shortcomings of the XGBoost algorithm, the LightGBM algorithm has been proposed. The XGBoost algorithm is optimized by the LightGBM algorithm in the following three aspects (Ke et al., 2017): (1) using a histogram-based algorithm to tackle the difficulty of too many split points in the CART tree; (2) adopting gradient-based one-side sampling (GOSS) to address the issue of having too many samples; (3) employing exclusive feature bundling (EFB) to resolve the problem of excessive features.

Through histogram statistics, large-scale data is placed in the histogram. The basic framework is shown in Fig. 1.

Compared with the XGBoost algorithm, which needs 32-bit floating-point numbers to store characteristic values, the histogram algorithm can generally use 8-bit integers to store the discrete values of characteristics.Therefore, the memory usage can be saved. The histogram only requires calculating the gain for k node splits, reducing the time complexity from O(Number of eigenvalues * features) to O(k * features), which significantly improves the running speed. A small gradient usually indicates a small training error. Training the model with data having small gradients does not improve the model’s accuracy but can reduce efficiency with only a slight increase in running time. However, directly discarding this portion of data with small gradients would alter the data distribution and reduce the model’s accuracy. The GOSS algorithm effectively addresses this issue. It samples according to the weight information, ensuring that the data distribution is not significantly changed while reducing a large number of samples with small gradients. Specifically, the algorithm selects a% of the data with larger gradients, then randomly selects b% of the data with smaller gradients, and multiplies the b% data by a coefficient (1-a)/b. This approach ensures the algorithm pays more attention to samples with small gradients without significantly altering the original dataset’s distribution.

The EFB algorithm is a lossless method for reducing feature dimensions. It operates by calculating the conflict ratio between different features to measure the degree of non-exclusivity. A low conflict ratio indicates a high level of mutual exclusivity. Features with high mutual exclusivity can be bundled and fused, thus reducing the number of features and improving operational efficiency.

DNN model

The essence of DNN is to build a network structure with multiple hidden layers based on massive data (see Fig. 2) to ultimately enhance model performance. In contrast to manual rule selection or the construction of important features, DNN can retain more information from the data and reduce dependence on feature engineering.

The two adjacent layers of neurons are fully connected, and the initial weights are generated through unsupervised pre-training. The output results are applied as the input data for the next hidden layer and are carried out successively until the final results are output; this process is known as forward propagation. In addition, the weight value of each hidden layer is obtained by error back-propagation.

In the forward propagation algorithm, the mapping relationship between the two connected layers can be written as follows: (11)${\displaystyle z=\sum _{i=1}^m{w}_i{x}_i+b}$ where m means the number of input neurons, x_i repressents the output data of the previous neuron, w represents the vector of weight, and b stands for the offset term. In order to prevent the output from increasing infinitely, an activation function σ(z) is added, such as tanx, softmax, ReLU and so on. At the same time, Eq. (11) is expanded. Assuming that there are m neurons in layer l − 1, the output of the jth neuron in layer l can be expressed as: (12)${\displaystyle a_j^l=\sigma \left(z_j^l\right)=\sigma \left(\sum _{k=1}^m{w}_{jk}^l{a}_k^{l-1}+b_j^l\right)}$

Assuming that there are n neurons in layer l, by extending Eq. (12) and using matrix method, the output of layer l can be obtained: (13)${\displaystyle {\sigma }^l=\sigma \left(z^l\right)=\sigma \left(W^l{a}^{l-1}+b^l\right)}$ where W^l represents the weight coefficient matrix with the size of n × m in the l layer, and b^l represents the offset vector with the size of n × 1.

The error back-propagation algorithm involves selecting a loss function to measure the output loss of samples during model training and minimizing this loss function. The corresponding results are the coefficient matrix

W and the offset vector b. In DNNs, the process of finding the optimal values is generally completed through iterations of the gradient descent method.

TabNet model

To enhance predictive power, traditional DNNs often blindly increase the number of network layers, leading to the over-fitting of the model. The TabNet model is a neural network structure with a decision manifold similar to that of a tree model. It has realized the process of calculating the gain of each feature separately when the tree splits, as shown in Fig. 3 (Arik & Pfister, 2021).

Next, Fig. 4 displays the complete TabNet encoder architecture. The original data are normalized through the Batch Normalization module and then processed in multiple sequential steps, such as STEP1, STEP2, etc.

The feature transformer module functions to extract features, enabling the extraction of more effective information representations for sample attributes. Then, the obtained information is divided into two parts by the SPLIT module. One part is used for the current output, and the other serves as the input for the next step.

Subsequently, the attentive transformer module learns the importance of each feature in each sample and obtains the corresponding mask matrix to filter out the unimportant features. Figure 5 shows the basic structure of this module. The formula based on Fig. 5 is expressed as follows: (14)${\displaystyle M\left[i\right]=\text{Sparsemax}\left(P\left[i-1\right]\cdot h_i\left(a\left[i-1\right]\right)\right)}$

In Eq. (14), a[i-1] is the characteristic information divided by SPLIT in the last step decision, h_i represents FC and BN layerS, P[i-1] is the priors scales item, Sparsemax is similar to softmax, which can obtain more sparse output results. Where, $P\left[i\right]={\prod }_{j=1}^i\left(\gamma -M\left[j\right]\right)$, is used to indicate the degree of use of features in the previous steps. When γ is smaller, the features selection is sparser.

According to the properties of Sparsemax, we can obtain: (15)${\displaystyle \sum _{j=1}^{D}M{\left[i\right]}_{b,j}=1}$

Therefore, M[i] can be understood as the attention weight distribution of the D-dimensional feature for each input sample at the current step of the model. For different samples, the output attention weight varies, a characteristic referred to as instance-wise. Based on this characteristic, the TabNet algorithm can select different features for different samples, an improvement over the tree-based model, which does not have this ability.

The TabNet algorithm inherits the advantages of DNN, including representation learning and end-to-end training, which greatly reduces the dependence on feature engineering. At the same time, it combines the advantages of tree-based methods, including strong interpretability and sparse feature selection, thereby improving accuracy and efficiency.

Data set introduction and analysis

The data adopted of this article contains the features of real historical data from thousands of investments, and the main fields include time_id, investment_id (3,587 different stocks), target (return on investment) and 300 anonymous features generated from market data. The investment_ id belongs to ID feature, and the others belong to numerical features. Before model training, these two types of features need to be processed with different encoding methods. Embedding algorithm has the ability to decrease the dimension of ID feature, while ensuring the integrity of its information; due to the low dimension of other numerical features, One-hot encoding can be adopted.

In addition, due to commercial confidentiality, data exists in the form of processing and encryption.

Through the statistical analysis of different stocks in the data, we can see that the time span of most stock data is about 1,200 days from Fig. 6, which is conducive to improving the prediction accuracy of the model with uniform distribution data.

Figure 7 shows the distribution of stock returns. It can be seen that the data are mainly concentrated in the area with small absolute values of returns, and the returns away from the x-axis occupy only a small amount of data. The distribution of features is consistent with the distribution of stock returns, as shown in Fig. 8, which indicates that the fifth feature has a positive correlation with stock returns.

Proposed model

(1) Model building

This article proposes a hybrid model based on the DNN and TabNet algorithms to calculate stock returns and prices. Figure 9 shows the flowchart of the hybrid model.

After encoding the ID and numerical features in the initial data so that their dimensions do not differ too much, the data are input into the DNN and TabNet models for training, respectively. Finally, the predictions from the two models are combined to yield more accurate values.

The DNN model is constructed using fully connected layers with the Swish activation function. The DNN structure used in this article is shown in Fig. 10.

After encoding and dimensionality reduction of two different types of data, we extract and learn the associations between features through three fully connected layers (dense layers). We combine the features learned above and then output the final result through three Dense layers. In addition, to enhance the robustness of the DNN model and reduce the probability of overfitting, Gaussian noise is added to the input numerical data, which has little impact on the predicted results.

The structure of the TabNet model has been introduced in detail in ‘Algorithm Principle and Structure’.

Like other neural network models, the TabNet algorithm is very sensitive to hyperparameters, so adjusting and selecting appropriate parameters is important. Generally speaking, the parameters that greatly impact TabNet include N_steps, feature_dim, gamma, lambda sparsity, etc. Among them, larger data and more complex tasks require larger N_steps, but this may lead to overfitting. Adjusting the values of feature_dim and output_dim is the most effective way to balance performance and complexity. Through adjustment and optimization, the selection of model parameters in this article is shown in Table 1.

PCC evaluation index

In order to quantitatively evaluate the performance of our model and others, the Pearson correlation coefficient (PCC) is selected as the evaluation index. This indicator measures the correlation between two variables, x and y, with its value ranging between −1 and 1. Generally speaking, the larger the PCC value, the closer it is to 1, indicating a greater positive correlation between the two variables. The calculation expression can be written as follows: (16)${\displaystyle {\rho }_{X,Y}=\frac{cov\left(X,Y\right)}{{\sigma }_X{\sigma }_Y}=\frac{E\left[\left(X-{\mu }_X\right)\left(Y-{\mu }_Y\right)\right]}{{\sigma }_X{\sigma }_Y}}$ where σ_X and σ_Y are the standard deviations of variables X and Y respectively.

RMSE evaluation index

Root mean square error (RMSE) index is a common index for regression task as shown in the following formula: (17)${\displaystyle \text{RMSE}=\sqrt{\frac{1}{\mathrm{n}}{\left(\mathrm{Y}-\hat{\mathrm{Y}}\right)}^{\mathrm{T}}\left(\mathrm{Y}-\hat{\mathrm{Y}}\right)}}$ where Y is the vector of ground truth and $\hat{\mathrm{Y}}$ is the prediction vector and n is the sample number.

Result analysis

The PCC between the predicted results of stock returns obtained by different models and the real values is calculated (see Table 2). The larger the PCC value, the closer the predicted result is to the real return. To some extent, this index can express the accuracy of the model’s predictions.

Obviously, our hybrid algorithm has the highest PCCs value of 0.1269, which is better than the separate DNN and TabNet models, which are 0.1103, 0.1205 respectively. Then, the tree-based algorithms are compared, with the PCC values of the XGBoost algorithm and the LightGBM algorithm reaching 0.1102 and 0.1145, respectively. The combination of the DNN algorithm and the TabNet model makes the prediction accuracy on tabular data or time series exceed that of tree-based algorithms such as XGBoost and LightGBM, which have traditionally held a dominant position. This indicates that the model we propose performs exceptionally well on large datasets with minimal feature engineering required. The deep learning method has become a powerful tool for enhancing the model’s performance in predicting stock prices and returns.

Except to PCCs metric, another metric RMSE for evaluating the accuracy of the proposed method. RMSE is a metric to evaluate the regression task and a smaller RMSE means a better performance. Thus, from Table 3, proposed model achieves a lowest RMSE score, which means beating other single models.

In addition, the TabNet model, based on the attention transformer and instance-wise processing, has a strong explanatory capability and enhanced learning ability. This is attributed to the selection of the most influential and significant features at each decision step. The model can quantify the contribution of each feature to the training model, as shown in Fig. 11. Among the given 300 features, the 164th, 22nd, and 61st features evidently have the greatest impact on the performance of our model. According to the results in Fig. 11, during model training, we can filter out the features that have little or no impact on the performance of the model and enhance the learning of the features that have a greater impact, thereby obtaining a more accurate prediction result.