An attention embedded DUAL-LSTM method for financial risk early warning of the three new board-listed companies

Attention

There are two methods to classify financial state based on LSTM hidden state: One is to take the final hidden state of LSTM as the input of the classifier, which will lose some information far from the final hidden state; the other is to sum the hidden states of all time steps and then take the average value as the input of the classifier, but it cannot distinguish the influence of the input information of each time step on the composition classification. Therefore, we decide to use the attention mechanism to learn the importance of the input information at each time step to capture the critical information in many financial states and increase the weight.

Regard the input vector as LSTM x₀, x₁, …, x_t, and the hidden state corresponding to the input, that is, vectors are h₀, h₁, …, h_t respectively. We then calculate the similarity S_jbetween the current hidden state h_j and the final hidden state $\tilde{\mathrm{h}}$ by using an attention mechanism. The formula is as follows: (11)${\displaystyle {\mathrm{S}}_{\mathrm{j}}=tanh\left({\mathrm{W}}_{\mathrm{n}}{\mathrm{h}}_{\mathrm{j}}+{\mathrm{b}}_{\mathrm{n}}\right)}$

where W_n and b_n are the linearization matrix and bias. At each time step, the weight a_j, which is to process input information, is shown in the following formula: (12)${\displaystyle {\mathrm{a}}_{\mathrm{j}}=\frac{exp\left({\mathrm{S}}_{\mathrm{j}}\times \tilde{\mathrm{h}}\right)}{\sum _{\mathrm{m}=0}^{\mathrm{t}}\left({\mathrm{S}}_{\mathrm{j}}\times \tilde{\mathrm{h}}\right)}.}$

The hidden state vector v of LSTM after adding attention weight is shown in the following formula: (13)${\displaystyle \mathrm{v}=\sum _{\mathrm{m}=0}^{\mathrm{t}}\left({\mathrm{a}}_{\mathrm{j}}{\mathrm{h}}_{\mathrm{j}}\right).}$

Dual LSTM embedded attention

Aiming at the financial feature representation x₀, x₁, …, x_t of enterprises listed on the New Third Board, this article proposes a novel dual LSTM to generate attended vector and warning classification results, respectively. The dual LSTM can be divided into the attention LSTM and the classification LSTM, respectively, whose basic framework is shown in Fig. 2.

In each time step, the attention LSTM includes the output of the particular classification LSTM at the previous time, the financial feature vector and the generated financial status encoding feature, respectively. The specific representation is shown in the following formula: (14)${\displaystyle {\mathrm{h}}_{\mathrm{t}}^1={\text{LSTM}}_{\mathrm{att}}\left({\mathrm{h}}_{\mathrm{t}-1}^2,{\mathrm{x}}_{\mathrm{t}}\right)}$

where ${\mathrm{h}}_{\mathrm{t}-1}^2$ is the output of the Classification-LSTM at the previous time step. These inputs provide the status information of the classified LSTM, the context information of the financial state and the maximum context information of the generated financial state, respectively.

After obtaining the output ${\mathrm{h}}_{\mathrm{t}}^1$ of the attention LSTM at time t. At each time step, the attention weight a_j,t is generated for each financial feature x_t: (15)${\displaystyle {\mathrm{a}}_{\mathrm{j},\mathrm{t}}={\mathrm{W}}^{\mathrm{T}}tanh\left({\mathrm{W}}_{\mathrm{x}}{\mathrm{x}}_{\mathrm{t}}+{\mathrm{W}}_{\mathrm{h}}{\mathrm{h}}_{\mathrm{t}}^1\right)}$

where W^T, W_x, W_h are learnable matrixes. Then, according to Eq. (8), the financial features ${\tilde{\mathrm{x}}}_{\mathrm{t}}$ are input to the classification LSTM. After attention financial features of LSTM are obtained, the classification results y are generated by combining the output of the classification LSTM.

(16)${\displaystyle {\mathrm{h}}_{\mathrm{t}}^2={\text{LSTM}}_{\mathrm{cls}}\left({\tilde{\mathrm{x}}}_{\mathrm{t}},{\mathrm{h}}_{\mathrm{t}}^1,{\mathrm{h}}_{\mathrm{t}-1}^2\right)}$ (17)${\displaystyle \mathrm{y}=\text{softmax}\left({\mathrm{W}}_{\mathrm{cls}}{\mathrm{h}}_{\mathrm{t}}^2+{\mathrm{b}}_{\mathrm{cls}}\right)}$

where W_clsandb_cls are learnable linear transformation matrix and bias, respectively.

The financial indicators adopted by the model training are shown in Table 1.

Comparison results of different models

To evaluate our proposed model in this article, several classical financial early warning models are selected for experimental comparisons, such as the Z-score model (Yan & Ma, 2011), Fisher discriminant method (Huang, 2021), logistic regression (Liguo et al., 2013) and BP network on the same dataset. We present the results as shown in Table 3.

Because this article focuses on the early warning of financial risk, the special treatment (ST) companies’ identification into the recalling rate of ST companies is even more critical. As seen above, the dual LSTM embedded Attention model has better performance, and the dual LSTM model can achieve a value of more than 90% in terms of recall, precision and F measure, which is a specific identification model of ST and the l performance of the classifier is better. First, the F value of the attention-embedded dual LSTM is increased by more than 18% compared with the original Z-score because all the model structures designed in this article are ahead. Secondly, our method improves the F value by more than 5% to the Fisher’s discriminant in all three evaluation indexes. Then, the effect of these two methods, Logistic and BP neural network, are significantly improved, which are still inferior to our proposed method. Finally, the proposed method achieves the best experimental results by embedding the Attention module and creatively using the two-layer LSTM as a financial warning.

Influence of hidden layer size and layers number

To discuss the number of LSTM and the size of hidden layers, we set the batch size to 4 and epochs =50 during training to experiment. In the experimental process, we will complete the determination of hyperparameters by comparing the recognition accuracy during training.

The dimension size of the hidden layer can be expressed as a bar graph for the model training results (Acc refers to the correct rate of recognizing ST into ST). As shown in Fig. 3, we can find that the model’s performance continues to rise with the increase of dimension until the hidden layer dimension reaches a peak of 1024, where the model achieves the best training accuracy. Therefore, we choose the dimension of the hidden layer to be 1024.

The number of layers can be represented as a histogram (Acc refers to the correct rate of recognizing ST into ST). As shown in Fig. 4, we can also find that the model’s performance continues to rise with the number of model layers increasing. When the length of the LSTM is 6, our method achieves the best results and training accuracy. So, we set the number of the model layer to 6.

The above results directly affect the training process’s rigor and positively affect the financial risk prediction of the Three New Board listed enterprises.

Comparison of prediction stability

To compare the predictive stability of our model with the Z-score model, Fisher discriminant method, logistic regression and BP neural network model Since the 1:1 of the rate can affect the randomness. This article changes the proportion between positive and negative samples by adding more positive samples corresponding to the negative samples one by one to investigate the robustness of the dual LSTM embedded Attention model. Specifically, the ratio between the financial risk sample and the safe financial sample is reset, following the ratio of 1:2, 1:3, 1:4, 1:5 and 1:6, respectively. Then, we apply these five proportions to structure the five types of models, which are to evaluate the prediction stability for the financial risks. Because the samples, which are chosen with the five proportions, are not balanced samples, we exploit a special measurement F for the non-balanced samples to obtain the performance of the prediction. The experiment results are presented in Fig. 5.

We can find that the F value of our proposed model is higher than other models with any proportions of positive and negative samples, which indicates that Z-score, Fisher discriminant, Logistic regression and BP network are inferior to our method for both financial risk samples and safe samples. Graphically, the F value of our proposed model is relatively more stable in trend and less volatile, which indicates that the prediction of the financial risk samples and safe samples are accurate with different proportions.