Explainable stock prices prediction from financial news articles using sentiment analysis

LSTM

LSTM (Long Short-Term Memory) is an improved form of RNN. LSTM models avoid the problems encountered by RNN. Hochreiter & Schmidhuber (1997) introduced LSTMs that make use of memory cells that can either forget unnecessary information or store information for more extended periods. LSTMs are explicitly modeled to handle tasks involving historical texts and are also able to educate themselves on long term dependencies. With the help of memory cells, they are capable of educating themselves. LSTMs have a chain-like structure making it easier to pass on information. The information is passed on as a state of the cell from one memory cell to another. The output of the network is modified by the state of these cells.

The architecture of LSTM allows for constant error flow with the help of constant, self-connected units (Hochreiter & Schmidhuber, 1997). This flow of error and states is propagated with the help of the three gates: input gate, output gate and forget gate, that each LSTM memory cell block is composed of. Input gates modulate the amount of new information received by a cell, forget gates determine what amount of information from the previous cell is passed on to the current cell; they determine what information is relevant and what information needs to be forgotten (Kim & Won, 2018). Given below in Fig. 1A is the structure of Simple Recurrent Network (SRN). As compared to SRN, LSTM-Cell given in Fig. 1B is different and has multiple gates.

LSTM model has two passes:

• Forward Pass, and

• Backward Pass

A Forward Pass

Let x^t be the input for time t, N be the quantity of LSTM blocks, M is the number of inputs. We then get the subsequent weights for LSTM layer:

•Input weights: W_z, W_i , W_f , W_o ∈ R^N×M

•Recurrent weights: R_z, R_i , R_f , R_o ∈ R^N×N

•Peephole weights: p_i , p_f , p_o ∈ R^N

•Bias weights: b_z, b_i , b_f , b_o ∈ R^N

Then the vector formulas for a vanilla LSTM layer forward pass can be written as: (1.7)${\displaystyle {\underset{¯}{z}}^{\mathrm{t}}={\mathrm{W}}_{\mathrm{z}}{x}^{\mathrm{t}}+{\mathrm{R}}_{\mathrm{z}}{y}^{\mathrm{t}-1}+\mathrm{bz}}$ block input (1.8)${\displaystyle {\mathrm{z}}^{\mathrm{t}}=g\left({\underset{¯}{z}}^{\mathrm{t}}\right)}$ (1.9)${\displaystyle {\underset{¯}{i}}^{\mathrm{t}}={\mathrm{W}}_{\mathrm{i}}{x}^{\mathrm{t}}+{\mathrm{R}}_{\mathrm{i}}{y}^{\mathrm{t}-1}+p_{\mathrm{i}}\odot c^{\mathrm{t}-1}+{\mathrm{b}}_{\mathrm{i}}}$ input gate (1.10)${\displaystyle {\underset{¯}{i}}^{\mathrm{t}}=\sigma \left({\underset{¯}{i}}^{\mathrm{t}}\right)}$ (1.11)${\displaystyle {\underset{¯}{f}}^{\mathrm{t}}={\mathrm{W}}_{\mathrm{f}}{x}^{\mathrm{t}}+{\mathrm{R}}_{\mathrm{f}}{y}^{\mathrm{t}-1}+p_{\mathrm{f}}\odot c^{\mathrm{t}-1}+{\mathrm{b}}_{\mathrm{f}}}$ forget gate (1.12)${\displaystyle f^{\mathrm{t}}=\sigma \left({\underset{¯}{f}}^{\mathrm{t}}\right)}$ cell (1.13)${\displaystyle {\underset{¯}{f}}^{\mathrm{t}}={\mathrm{W}}_{\mathrm{f}}{x}^{\mathrm{t}}+{\mathrm{R}}_{\mathrm{f}}{y}^{\mathrm{t}-1}+p_{\mathrm{f}}\odot c^{\mathrm{t}-1}+{\mathrm{b}}_{\mathrm{f}}}$ (1.14)${\displaystyle {\mathrm{f}}^{\mathrm{t}}=\sigma \left({\underset{¯}{f}}^{\mathrm{t}}\right)}$ (1.15)${\displaystyle {\mathrm{c}}^{\mathrm{t}}=z^{\mathrm{t}}\odot i^{\mathrm{t}}+{\mathrm{c}}^{\mathrm{t}-1}\odot {\mathrm{f}}^{\mathrm{t}}}$ (1.16)${\displaystyle {\underset{¯}{o}}^{\mathrm{t}}={\mathrm{W}}_{\mathrm{o}}{\mathrm{x}}^{\mathrm{t}}+{\mathrm{R}}_{\mathrm{o}}{\mathrm{y}}^{\mathrm{t}-1}+{\mathrm{p}}_{\mathrm{o}}\odot {\mathrm{c}}^{\mathrm{t}}+{\mathrm{b}}_{\mathrm{o}}}$ (1.17)${\displaystyle {\mathrm{o}}^{\mathrm{t}}=\sigma \left({\underset{¯}{o}}^{\mathrm{t}}\right)}$ Output gate (1.18)${\displaystyle {\mathrm{y}}^{\mathrm{t}}=\mathrm{h}\left({\mathrm{c}}^{\mathrm{t}}\right)\odot {\mathrm{o}}^{\mathrm{t}}}$

where σ, g and h are pointwise non-linear activation functions. The logistic sigmoid (σ(x) = 1∕1 + e ^−x) is employed as an activation function and also the tangent hyperbole (g(x) = h(x) = tanh(x)) is typically used because the block input and output activation function. Pointwise multiplication of 2 vectors is denoted by ⊙ (Greff et al., 2015) .

B. Backpropagation

Calculation of deltas in LSTM Block: (1.19)${\displaystyle \delta {\mathrm{y}}^{\mathrm{t}}={\Delta }^{\mathrm{t}}+{\mathrm{R}}_{\mathrm{z}}^{\mathrm{T}}{\delta }_{\mathrm{z}}^{\mathrm{t}+1}+{\mathrm{R}}_{\mathrm{i}}^{\mathrm{T}}{\delta }_{\mathrm{i}}^{\mathrm{t}+1}+{\mathrm{R}}_{\mathrm{f}}^{\mathrm{T}}{\delta }_{\mathrm{f}}^{\mathrm{t}+1}+{\mathrm{R}}_{\mathrm{o}}^{\mathrm{T}}{\delta }_{\mathrm{o}}^{\mathrm{t+1}}}$ (1.20)${\displaystyle \delta {\underset{¯}{o}}^{\mathrm{t}}=\delta y^{\mathrm{t}}\odot \mathrm{h}\left(c^{\mathrm{t}}\right){\sigma }^{\prime }\left({\underset{¯}{o}}^{\mathrm{t}}\right)}$ (1.21)${\displaystyle \delta {\mathrm{c}}^{\mathrm{t}}=\delta {\mathrm{y}}^{\mathrm{t}}\odot {\mathrm{o}}^{\mathrm{t}}\odot {\mathrm{h}}^{\prime }\left({\mathrm{c}}^{\mathrm{t}}\right)+{\mathrm{p}}_{\mathrm{o}}\odot \delta {\underset{¯}{o}}^{\mathrm{t}}+{\mathrm{p}}_{\mathrm{i}}\odot \delta {\underset{¯}{i}}^{\mathrm{t+1}}+{\mathrm{p}}_{\mathrm{f}}\odot \delta {\underset{¯}{f}}^{\mathrm{t+1}}+\delta {\mathrm{c}}^{\mathrm{t}+1}\odot {\mathrm{f}}^{\mathrm{t}+1}.}$

Here, Δt is the vector of deltas delegated from the layer above. If E is the loss function, it corresponds to ∂E ∂yt , but not counting the recurrent dependencies (Pathmind, Inc., 2020). Then: (1.22)${\displaystyle \delta {\underset{¯}{f}}^{\mathrm{t}}=\delta {\mathrm{c}}^{\mathrm{t}}\odot {\mathrm{c}}^{\mathrm{t}-1}\odot {\sigma }^{\prime }\left({\underset{¯}{f}}^{\mathrm{t}}\right)}$

Why LSTM?

LSTM networks various state cells. These short and long-term memory cells rely on the state of these cells. These memory cells act as an aide for the model to remember historical context as predictions made by the network are influenced by past experiences of inputs to the network. This helps us make better predictions. LSTM networks tend to keep the context of information fed by inputs by integrating a loop that allows information to flow, in one direction i.e from one step to the following.

Explainable AI (XAI)

We want our model to output not only the prediction part but also the explanation as to why the prediction turned out that way. If our machine makes a prediction, why should the user trust the prediction made by the machine? Today, machine learning and Artificial Intelligence(AI) are exploited to make decisions in many fields like medical, finance, and sports. There are cases where machines aren’t 100% accurate. So, the user should blindly trust the choice of the machine? How can the user trust AI systems that derive inferences on probable unfair grounds? To solve the problem of trust between the user and Artificial Intelligence, XAI (Saffar, 2019) can be employed. It gives us the reasoning for a prediction made by the model. Mainly XAI is employed to resolve the black box problem. This ”black box” phase is interpreted by XAI and explained to the users. The user cannot completely depend on the model without a clear understanding of the model and the way the output is achieved. XAI provides a clear understanding of how the model achieved a certain prediction or result. XAI gives a human-understandable explanation for the prediction. Current models enable interpretation but leave it to the user to apply their knowledge, bias and understanding.

System design

Figure 2 explains the overall architecture of the system. We preprocess the News Headlines Dataset by performing tokenization, removing stop words, and embedding it. The headlines are then normalized and sentiment analysis is performed on it to comprehend the sentiment behind each sentiment, i.e whether a particular headline has a positive or negative sentiment associated with it. We then use the preprocessed headlines to classify them. We classify them to analyze whether they produce a positive sentiment or a negative sentiment. Later, this headline dataset, along with the preprocessed Yahoo Finance dataset, is combined to form the final dataset. Then the dataset is divided into train and test dataset. The training dataset is used to build the LSTM model. The test dataset to verify the results obtained after training. Finally, XAI is implemented using the LIME tool that is used to interpret the model and understand the biases involved in the dataset.

Prediction

XAI

Procedure

Data preprocessing

The data obtained is raw and unprocessed. To perform any computation, processing the data is necessary. For instance, the null data should be removed, and the trivial values should be removed, etc. Given below in the Fig. 3 is the raw dataset obtained from the sources.

This is what the Financial News dataset looked like before preprocessing. Table 1 shows the features neccessary for our model and we clean the dataset to obtain only these features.

Table 1:

Features of Clean data.

Final columns that are used in the dataset that is necessary for the model building and model evaluation.

Column A	Column B	Column C
News headline	Website	Timestamp

DOI: 10.7717/peerjcs.340/table-1

Steps involved in Data Preprocessing are represented in Fig. 4:

As we are first training the LSTM model only for BSE-SENSEX and NSE-NIFTY, we train it on news headlines involving SENSEX and NIFTY specific news only. For this, we search for headlines that contain the word: “Sensex” (case - independent).

After running the code on our cleaned dataset we get this result as shown in Fig. 5.

Sentiment analysis for stock market news headlines

To be able to predict the stock market (Patel et al., 2014), we must understand the market’s sentiment correctly. Negative news will lead to a fall in the price of the stock and positive news will lead to rising in the price of the stock. The most commonly used words in a news headline will give rise to an instant trigger of emotions in a particular person’s mind. Hence we made a word cloud of the top 150 words commonly occurring in a news headline from our dataset as depicted below in Fig. 6.

LSTM network is a type of RNN that preserves its state. (Olah, 2020) LSTM (Selvin et al., 2017) has already provided satisfactory results in the area of text data applications such as machine translation and NLP (Bird, Klein & Loper, 2016). The headlines are a type of text data; for that reason, it was a rational decision to apply this type of network to our data. Figure 7 shows the LSTM network architecture used in our project.

LSTM - CNN model for news headlines

1. Initially, we have created an instance of ReduceLROnPlateau, which reduces the learning rate when a metric has stopped improving accuracy.

2. Secondly, we have a ModelCheckpoint callback. It is used for training the model using model.fit() to save a model or weights at some interval, which can be loaded later to continue the training from the state saved.

3. An instance of EarlyStopping is created. This stops training the model when a particular metric has stopped improving. Then the preprocessed data is loaded and only the top n words are kept, and the rest are zero. Also, split it into the train dataset and test dataset. The data is then truncated and we pad the input sequence (X_train and X_test). A split of 70-30 is done for the train and test.

4. Finally, we create a model.

   1. The first layer is constructed using the sequential network model of the Keras framework. A Sequential model works good enough for vanilla stack layers. Each layer has one input tensor and one output tensor exactly.

   2. The next layer, i.e., an Embedding layer, is added. Here, the text data is encoded first. In an embedding, words are defined by dense vectors. Each vector serves the projection of the word into a continuous vector space.

   3. Then a MaxPooling1D layer is added. A pooling layer is an elementary constituent unit of CNN. The main function of the pooling layer is to sequentially minimize the spatial orientation size and the number of parameters and hence reduce the computation required over the network. The pooling layer operates separately for each feature map.

Out of all the pooling layers, the one best suited for our model was max pooling. It is depicted as follows: It takes the maximum value over the window defined by pool_size and hence downsamples the input representation. In our model, the pool_size is 2. Figure 8 visualises the working of CNN MaxPooling1D.

Next, we then construct an LSTM layer of 100 units, as our inputs’ maximum review length is 100. We then add a Dense layer having the activation function of sigmoid as this is our final layer. We also are using an adam optimizer with a learning rate of 1e⁻³. The model is finally compiled and then we save the model. We then print the model summary and the function is called for today and tomorrow parameters.

LIME

As represented in Fig. 9, after training the model and obtaining predictions the explainer interpretes the model so that the user can make appropriate judgements. After studying various XAI (Khaleghi, 2019; Ribeiro, Singh & Guestrin, 2016) tools like LIME, What-If (Wexler et al., 2019), DeepLift (Shrikumar et al., 2016), Shapely (Messalas, Christos & Yannis, 2019), we realized the one which will determine the Transparency of our model in the best possible way was LIME.

Table 2:

Hyperparameters, loss function and optimizer and their values used along with detailed comments.

Title	Value	Comments
Loss function	Binary cross-entropy	Since the type of classification performed is binary
Optimizer	Adam optimization	Used to enhance the network
Learning rate	0.001	Set after trials
Decay	0.1	Validation loss does not ameliorate significantly any more for 5 epochs, there will be decay.

DOI: 10.7717/peerjcs.340/table-2

According to the study, the What-if Tool is an interactive visual tool to help the user comprehend the datasets and better understand the output of TensorFlow models. To analyze the models deployed. However, it is well suited for XGBoost and SciKit Learn Models.

DeepLift is a tool that collates the activation of a neuron to its ’reference points’ and assigns contribution scores accordingly. A single backward pass is used to compute scores efficiently. DeepLift reveals dependencies that are otherwise missed by other methods giving separate consideration for positive and negative contributions.

LIME, short for Local Interpretable Model-Agnostic Explanations (Ribeiro, Singh & Guestrin, 2016), is a tool that is used to explain what the machine learning model is performing. It is applied to the models which perform behaviour but we do not know how it works.

Machine learning models (Patel et al., 2015) can be explained using plots, for example, linear regression, decision trees; these are some of the classifiers that can be explained using plots (It can only give an explanation for the global, but it cannot give an explanation of any local observation). But there are some significant data with more complexities and more dimensions that cannot be explained using plots. One of the examples is the neural network; it cannot be explained using plots.

By using LIME, we can understand what the model does, which features it picks on to create or develop prediction. LIME is model-agnostic; it means that it can be used for any black-box model that we know today or can be developed in the future. LIME tool is local; it means it is observation specific; it explains every single observation that you have in a dataset. If a model has a high accuracy, can we trust the model without having interpretability? The answer is no; we cannot trust it because many models have noises that can predict the output right, but the way the model has predicted will contain some faults which are not good for a long term process. So, interpretability is important for building trust with the model.

Performance measure

Result

Table 3 represents the results generated after performing the experiments.

In LSTM-CNN the input was the data that we get from preprocessing. The data was then combined with a dataset of the news headlines and the prices. The model which we trained was of the news headlines of Sensex.

There are a total of 100 epochs to be run in the training of the model. But it stops at the 15th epoch; this is due to avoiding making the model overfitting. The accuracy of the model in a total of 15 epochs is 74.76%. The Loss was 0.1693.

As we see in Fig. 10:

Epoch 00015:val_acc did not improve from 0.73876.

There are a total of 100 epochs. They stopped training at 15 to 18th epochs. This is because of the function EarlyStopping that helps to avoid the problem of overfitting.

The accuracy remains constant after the 15th epochs, whereas, val_acc starts decreasing due to which training stops.

val_acc:- It is the measure of how good are the predictions of the model. So, in our case, the model is very well trained till the 15th epoch, and after that, the training is not necessary.

acc:- It gives the percentage of instances that are correctly classified. In our case, the percentage is 93.15%.

val_loss:- val_loss is the loss that is applied to the testing set, whereas, the loss is applied to the training set. val_loss is the best to depend on to avoid the problem of overfitting. Overfitting is when the model is fit too closely to the training data.

The val_loss and loss keep on decreasing, whereas acc and val_acc are kept on increasing this shows that our model is well-trained. Figure 11 visualises the loss over number of epochs.

As according to the plot obtained after training, we observe that initially, the loss is very high; however the validation loss is low relatively. Once the epochs progress, the loss decreases, thus providing better learning for our model. The point where the loss and validation loss plateaus are the point where no further learning might be possible with the same amount of data for the same number of epochs. Although the accuracy of LSTM-CNN with news headlines data is less than LSTM with OHLC data, since we are interpreting the model, it is adding another perspective of viewing at the model, which makes it a better model. When compared with RNN-CNN (Tan et al., 2019; Zhang, Chen & Huang, 2018) model , which it performs way better since not only does it remember small term information but also long term information.

Results of XAI

As for the first model,as shown in Fig. 12 , we get a cumulative 1,000 explanation features. The word surge is mostly related in a negative context and hence is having a negative weight. Similarly, higher is mostly in those sentences which depict the growth of stocks and hence is having positive weight. We can also observe the word Sensex having near about neutral weight as it has both positive and negative references. Thus depicts both falls as well as the rise of stocks. By Table 4, we can thus interpret our data and thus understand the biases in the dataset. Our model is not just a black box now, and the customers know the insides of the system through just one graph. Thus we achieve the explanations along with the insights from the data.

Test results

As shown in above Fig. 13, the trained model can now be tested for predictions. The blue bar represents the predicted value, and the red bar represents the actual value obtained from the dataset. This gives us a fair understanding of the output. If we want to test for the real-time news, we can enter news that gets preprocessed and get a measure of the opening price, i.e., how much did the opening price rise or fall from the previous opening price. We present values in Table 4 for getting the exact idea of these values and gauge the accuracy of our predictions.

The table below shows the following: date of actual open , previous closing price, predicted opening price by our model, actual opening price, rise / fall in price, error percentage, chi-squared test.

Here the:

Rise/fall is given by: (Predicted Open -Previous Close) (1.23)

Error percentage is given by: abs((Actual-predicted)/(Actual ))*100 (1.24)

Chi-square is given by:(actual - predicted)²/predicted (1.25)

As presented above in Table 5, seven test cases have been stated with the stock market prices along with the rise and fall of those values. The error represents the value difference in percentage (Actual-predicted by our model) and for our test cases, it’s showing in the range 0.15 to 0.73. Thus, we could manage to predict the Indian stock market price using XAI by referring to the impact of financial news articles. Also the chi-square test allows us to assess how much is the observed data varying from actual data.