STTM: an efficient approach to estimating news impact on stock movement direction

Data preprocessing

First of all, we preprocess input text data. Each text is subject to tokenization and lemmatization, removal of stop-words, and punctuation symbols. Next, we calculate idf-parameter—inverse document frequency, part of TF-IDF feature, for each unique word w in D: (1)${\displaystyle \mathrm{idf}\left(w,D\right)=log\frac{\left|D\right|}{\left|\left\{d_i\in D\mid w\in d_i\right\}\right|},}$ where |D|—number of documents in collection; |{d_i ∈ D∣w ∈ d_i}|—number of documents from D collection, where the word w occurs. After that, we remove words in the upper and lower quantiles of the α-level from the text data.¹ Thus, we do not consider the most rare or the most frequent words. Then, we transform the resulting text data into a bag-of-word representation.

Topic modeling

We feed the preprocessed text data as an input to the topic model for generating probabilistic topics: T₁, …, T_n. The topic model can be LDA, DTM, DIM, ITMTF or any other technique.² It can be pre-trained in advance or online-trained on the textual data stream D. As a result of topic modeling procedure, each document d is represented by n - dimensional vector of topics’ probabilities: θ^(d) = (θ_d,1, …, θ_d,n). We numerically estimate the salience of each topic T_j in each time unit t_i of the textual data stream D as follows: (2.1)${\displaystyle {\Theta }_i^j=\sum _{\forall d\text{in}{t}_i}{\theta }_j^d.}$ Topic stream TS_j of each topic T_j is thus defined as a set of all salience scores of this topic in a given time period: (2.2)${\displaystyle T{S}_j={\Theta }_0^j,\dots ,{\Theta }_N^j.}$ Consequently, we associate each topic T_j with the time series of its stream TS_j.

Topic tonality based on word-level tone

By analogy with topic stream, to define word stream, for each word w we first calculate its frequency as the sum of its occurrences over all documents d that have appeared in our stream in a given time point t_i: (3.1)${\displaystyle c_i^w=\sum _{\forall d\text{in}{t}_i}c\left(w,d\right).}$ Thus word stream WS_w is defined as a set of word frequencies in a given time period: (3.2)${\displaystyle W{S}_w=c_0^w,\dots ,c_N^w.}$ Consequently, we associate each word w with the time series of its stream WS_w. In general, c(w, d) can be any additive function of the number of words w in the document d. The tone of the word w is determined as a function of target time series and word stream: (4.1)${\displaystyle {\Omega }_w=f_w\left(p_t,W{S}_w\right).}$ Function f_w can be any regression evaluation metric or any time series proximity metric. We use the Pearson correlation coefficient: (4.2)${\displaystyle f_w=r_{p_t,W{S}_w},}$ if the significance less than γ³ and (4.3)${\displaystyle f_w=0,}$ in other cases. Since each topic T_j is a probability distribution in each time unit t_i over a dictionary V: $T_{j,i}=\left(w_1,{\varphi }_{w_1}^{j,i}\right),\dots ,\left(w_{\left|V\right|},{\varphi }_{w_{\left|V\right|}}^{j,i}\right)$, we define the topic tone as a function of the word’s probabilities in the topic ${\varphi }_W^{j,i}$ and the corresponding word’s tone Ω_W for each time point t_i: (5.1)${\displaystyle {\psi }_{j,i}=f_{T_{j,i}}\left({\varphi }_W^{j,i},{\Omega }_W\right),}$ The overall tonality of topic T_j is defined as a set of its tones in a given time period: (5.2)${\displaystyle {\Psi }_{T_j}={\psi }_{j,0},\dots ,{\psi }_{j,N}.}$ We implement function f_Tj,i as follows:

1. The number of the most probable words in the topic T_j at time point t_i (i.e., the words for which the tone will be calculated) is selected so that the sum of their probabilities does not exceed the specified threshold C_j:⁴ (5.3)${\displaystyle \sum _{w\text{sorted by probability}}{\varphi }_w^{j,i}\le C_j,}$

2. The variables pProb and nProb are calculated. The calculations involve only words selected in the previous step: (5.4)${\displaystyle \text{pProb}=\frac{\sum _{{\Omega }_w\ge 0}{\Omega }_w{\varphi }_w^{j,i}}{\sum \left|{\Omega }_w{\varphi }_w^{j,i}\right|},\text{nProb}=\frac{\sum _{{\Omega }_w<0}\left|{\Omega }_w{\varphi }_w^{j,i}\right|}{\sum \left|{\Omega }_w{\varphi }_w^{j,i}\right|}.}$ If the significance level r_pt,WSw of all the selected words is higher than γ, then we define: (5.5)${\displaystyle \text{pProb}=0,\text{nProb}=0.}$ The final topic tonality is expressed as the difference between pProb and nProb: (5.6)${\displaystyle f_{T_{j,i}}=\text{pProb}-\text{nProb}.}$

Tonal topic stream

At the next step, we calculate the tonal topic stream (TTS) as a product of topic tonalities and topic streams for each time point: (6)${\displaystyle TTS=\left[\begin{array}{ccc}\hfill {\Theta }_0^0\cdot {\psi }_{0,0}\hfill & \hfill {\Theta }_1^0\cdot {\psi }_{0,1}\hfill & \hfill \dots \hfill \\ \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill \hfill \\ \hfill {\Theta }_0^n\cdot {\psi }_{n,0}\hfill & \hfill \hfill & \hfill {\Theta }_N^n\cdot {\psi }_{n,N}\hfill \end{array}.\right]}$ Thus TTS is a matrix, where rows correspond to the topics and columns correspond to the time points. Each element of this matrix, TTS_j,i, is the value of tonal topic stream for the topic T_j at time point t_i.

STTM index

Finally, STTM index as the overall tonality of all topics over a given period of time is defined as a time aggregate from the TTS matrix: (7)${\displaystyle \text{STTM}={\text{aggregate}}_t\left(\text{TTS}\right).}$ The aggregation function can be a simple or weighted mean, median, or sum. We use simple sum by default. The tonality of each specific news item d is, analogously, the aggregate over the products of topic probabilities of news item θ^(d) and its topic tonalities ψ^(d) at time point t_d. As noted above, for comparability purposes we normalize the STTM index to the range [0, 1] based on sigmoid regressor.

Figure 2 shows a general scheme for predicting directions of financial market movements using the STTM approach.

MOEX Russia Index

MOEX (Moscow Exchange) Russia Index (see Fig. 3) is a capitalization-weighted composite index serving as the primary ruble-denominated benchmark of the Russian stock market. It is calculated as the sum of the prices of 39 most liquid Russian stocks weighted by expert assessments of their impact on the Russian economy. These stocks are pre-selected by experts from among the largest and the most dynamically developing Russian issuers with economic activities in the leading sectors of the Russian economy. MOEX Russia Index is used as one of the baseline investment portfolios in this research. The shares time series from 2013 to 2021 included in MOEX Russia Index (available at: https://www.moex.com/en/index/IMOEX) listing constitute our times series dataset analyzed in this article.

There is 39 main time series with the following tickers:

SBER, SBERP, GAZP, LKOH, YNDX, GMKN, NVTK, SNGS, SNGSP,

TATN, TATNP, ROSN, POLY, MGNT, MTSS, FIVE, MOEX, IRAO,

PLZL, NLMK, ALRS, CHMF, VTBR, RTKM, PHOR, TRNFP, RUAL,

AFKS, MAGN, DSKY, PIKK, HYDR, FEES, QIWI, AFLT, CBOM,

LSRG, RSTI, UPRO.

Each time series of the mentioned shares is converted to the weekly returns r_t:

$r_t\equiv \frac{p_t-p_{t-1}}{p_{t-1}}$,

where p_t is the closing price of the shares time series, t - weekly timestamp.

Economic news

Our text dataset of daily news includes three Russian national media sources: Kommersant, RIA Novosti, and Vedomosti. Kommersant (The Businessman, available on the website: https://www.kommersant.ru) is a nationally distributed daily newspaper devoted to politics and business. RIA Novosti (Russian Information Agency, available on the website: https://ria.ru) is one of the principal state-owned news agencies publishing news and opinions of social, political, economic, scientific, and financial subjects. Vedomosti (The News, available on the website: https://www.vedomosti.ru) is a national daily newspaper specializing in business. In each media outlet we consider the texts from the economy section only. Consequently, it contains a significant number of editorials, analytical reviews, and expert opinions, affecting the estimated textual data flow. Table 1 illustrates the amount of collected data and the date intervals corresponding to it. Figure 4 shows the main statistics about the Kommersant newspaper. The top panel of the figure plots the histogram of the number of articles (ordinary news and analytical reviews) vs the number of characters. The bottom panel shows the distribution of the weekly number of articles over eight considered years. The drops in the number of news correspond to public holidays and weekends. The graphs for the other daily sources can be found in Figs. B1 and B2.

News preprocessing pipeline

We use a common natural language preprocessing pipeline. We begin with the tokenization⁵ by breaking each text into sentence components and then into word components. Next, we normalize all tokens in each article to lower case letters, remove punctuation, non-alphabetic, and non-Cyrillic symbols, and perform lemmaization with Yandex MyStem—an instrument developed specially for the Russian language and based on extensive morphological analysis.⁶ A lemmatizer was preferred to stemmers because it avoids aggressive suffix-stripping—an approach that would not be applicable for highly inflected languages, such as those of the Slavic family, since therein suffixes are heavily used for word formation and can entirely change word meaning. In terms of recognizing lemmas from word forms, Yandex MyStem has the error rate of about 2–3% only and consistently outperforms other existing models on different Russian-language corpora (Kotelnikov, Razova & Fishcheva, 2018). Finally, we remove stop words, such as prepositions, participles, interjections, numbers⁷ and the tokens in the upper and lower quantiles of the α-level idf-parameter. We use the standard 95%-quantile as a default value for α-level. Finally, we convert each news item into a vector of word counts.

Granger causality test

The Granger causality test⁸ is a statistical hypothesis test for determining whether one time series is useful in forecasting another. Granger causality requires time series stationarity. Let y_t and x_t be two time series. To see if x_t ’Granger causes’ y_t with maximum q time lag, the following regression is performed: (8)${\displaystyle y_t=a_0+a_1{y}_{t-1}+...+a_q{y}_{t-q}+b_1{x}_{t-1}+...+b_q{x}_{t-q}.}$ Then, F-tests are used to evaluate the significance of the lagged x terms. The coefficients of lagged x terms estimate the impact of x on y. The null hypothesis that x does not Granger-cause y is accepted if and only if no lagged values of x are retained in the regression. Since in our case the time series has shown non-stationary behavior, as determined by the Dicky-Fuller unit root test,⁹ a first-order and, where necessary, a second-order differentiation was performed to achieve stationarity.

Sharpe ratio and annual return

The Sharpe ratio¹⁰ measures the performance of an investment, such as a share or a portfolio of shares compared to a risk-free asset, after adjusting for its risk. It is defined as the difference between the returns of the investment and the risk-free return, divided by the standard deviation of the investment returns. It represents the additional amount of return that an investor receives per unit of increase in risk. More formally, (9)${\displaystyle \text{Sharpe ratio}=\frac{R_p-R_f}{{\sigma }_p},}$ where R_p - return of portfolio, R_f - risk-free rate, σ_p - standard deviation of the portfolio’s excess return. As a risk-free asset for the Russian market, we use government bond yields (available on the website: https://www.cbr.ru/eng/hd_base/zcyc_params/). A Sharpe ratio of less than one is usually considered unacceptable or bad. It means that the risk of portfolio encounters is being offset well enough by its return.

Annual return or Compounded Annual Growth Rate (CAGR)¹¹ is the average annual rate calculated from the returns observed in the first and the last years of a given time span, assuming that all the dividends are reinvested in the end of each year. More formally, (10)${\displaystyle \text{Annual return}={\left(1+\frac{EV-SV}{SV}\right)}^{\frac{1}{n}}-1,}$ where EV - ending value, SV - start value, n - number of years.

STTM

As mentioned before, our Stock Tonal Topic Modeling (STTM) approach can have any basic topic model as its core. In this article, we implement two models: LDA¹² and DTM¹³ with python-wrapper from gensim python-package: https://radimrehurek.com/gensim/. Qualitative analysis of these models is presented in the section ‘Qualitative Analysis of Russian Economic News Topic Modeling’ in Appendix A. DTM extends LDA by allowing word probabilities in a topic to change over time. We have chosen to update them in the increments of one month. Further, we select the number of topics so as to avoid solutions with either a large number of too granular topics or a small number of too broad topics. To do so, we optimize topic coherence with Roder’s C_v metric (Röder, Both & Hinneburg, 2015). Although the dynamics of coherence change with the increase of the number of topics is somewhat different for different news sources (see Figs. B4, B5 and B6), the overall optimum appears at n = 20, where C_v curve reaches its maximum before flattening out for all national media sources. This optimum is the same for both LDA and DTM. We run a topic model on the training dataset and apply it to the test dataset. The datasets are constructed according to the procedure described in ‘Evaluation procedure’ section below. Topic tonality based on word-level tone is calculated from the solution obtained on the training set. Topic stream is calculated from the the test set, and topic tonality based on word-level tone is applied to it. STTM hyper-parameters are selected based on optimization of ROC-AUC metrics the grid-search on the training set for each time series independently. Examples of the STTM index, the stock time series, news, topics, words and also their tonalities and tones, as well as an example of a tonal topic stream are presented in Figs. B9, B10, B11 and B12. It can be seen that results of proposal procedure are highly interpretable. Since topic modeling possesses a certain level of instability leading to fluctuations in the word probabilities, we repeat all calculations for trading strategy performance at least ten times. After that we estimate the mean and variance of each of the considered economic metrics.

SESTM

We implemented the Sentiment Extraction via the Screening and Topic Modeling (SESTM) procedure (Ke, Kelly & Xiu, 2020) as a baseline topic model. We apply it for each time series from the MOEX Russia Index for three national news sources. SESTM approach infers only two topics—containing words that have either negative or positive effect on the target time series, the composition of these topics being optimized iteratively based on an association metric. The process consists of three steps: 1. isolating a list of sentiment terms via predictive screening 2. assigning sentiment weights to these words via topic modeling 3. aggregating terms into an article-level sentiment score via penalized likelihood. The main assumption of the model is that the news is generated from the following mixture multinomial distribution: (11)${\displaystyle d_{i,\left[S\right]}\sim \text{Multinominal}\left(s_i,p_i{O}_{+}+\left(1-p_i\right)O_{-}\right),}$ where s_i is the total count of sentiment-charged words in article d_i, p_i is the sentiment score, O₊ and O₋ are a positive and negative topics, respectively, which is probability distributions over words. The objective of SESTM procedure is to learn the model parameters O₊, O₋ and p_i. SESTM algorithm has three hyper-parameters of screening for sentiment-charged words and one hyper-parameter for regularization in learning-optimization problem. All hyper-parameters are tuned through the time cross-validation procedure with l¹-norm of the differences between estimated article sentiment scores and the corresponding standardized return ranks as a loss function. (Figures B13 and B14) in Appendix B contain examples of the most common words with cumulative positive (negative) tones, and the most cumulatively positive (negative) tonal words for VTB (VTBR) share prices. It can be seen that the contents of these topics is mixed and broadly uninterpretable.

Shallow feature based methods of text processing

To compare our approach to the models with shallow features based methods of text preprocessing, we apply a specific pipeline shown on Fig. 5. For each economic news from the considered news agencies (Kommersant, Vedomosti, RIA Novosti), various textual components are extracted (full text, first paragraphs, titles). After that, each component is preprocessed as described in section ‘Datasets and preprocessing’. Further, various techniques for obtaining embeddings are applied to each news item: Word2Vec, Navec, Doc2Vec, FastText,¹⁴ Navec realization from natasha python-package: https://github.com/natasha/navec, FastText realization as python-package from: https://fasttext.cc/ (all embeddings models trained on the first two years of texts for each news outlet separately). The resulting embeddings of news (where all the news for the same week are treated as one document) are fed as the input features to the following machine learning models: Random Forest (RF), Logistic Regression (LR), Gradient Boosting Machine (GBM), Support Vector Machine (SVM), and 3-layers Neural Network (NN).¹⁵ The target variable for all models is the sign of returns for each ticker included in MOEX Russia Index, which equals 1 for the growing times series and 0 otherwise.

Endogenous models

In addition, we compare the proposed framework with simple endogenous models: Random forest (RF), logistic regression (LR), gradient boosting machine (GBM), support vector machine (SVM), and three-layers neural network (NN), where weekly return lags are used as features, and the target variable is the same as in shallow-feature-based models. Figure 6 shows the pipeline we used for construction of such endogenous models.

Evaluation procedure

As for time series data, test and train datasets have to be defined as subsets covering uninterrupted periods of time, we use an iterative expanding cross-validation scheme (visualised in Fig. 7) in which we expand the time window of the training set by one year at each iteration, starting from two years and ending with six years, while test set window is kept stable at the length of one year across all iterations. Topic model obtained on the training set is retrained at each iteration and then applied to the test set. We estimate our models on all economic news between stock market start time each Monday and its end time each Friday. Considered target variable is movement direction sign(r_t) between the closing price on Friday and the opening price on the previous Monday for each share in the MOEX Russian Index.

For the proposed STTM approach, we compute a Granger causality test between the weekly value of STTM index and the weekly stock price of each ticker included in MOEX Russia Index for each test interval separately. For each ticker and for different topic models (LDA and DTM), including STTM, we evaluate the weekly trading strategy performance. We use a straightforward long strategy. For this trading strategy each Friday at the time of the closure of the stock market we select top 20 percent of stocks with the highest value of the model prediction for the current week. These stocks are the most likely to demonstrate price gains in the upcoming week. Next, we buy these 20% of stocks at the Friday prices. Such procedure is repeated each Friday thus providing portfolio recomposition. We then evaluate this strategy with the annual returns and Sharpe ratio metric introduced before. In doing so, we do not account for either broker commission or transactions costs (see Limitations section) because our goal is to evaluate the predictive power of our method as compared with other methods, rather than to calculate the amount of the final return it allows to gain.

Granger causality tests

In this section, we provide numerical results for the Granger causality test between STTM index and Friday’s prices for each of 39 tickers included in MOEX Russia Index; this is done for two topic models employed (LDA and DTM) and for three different news sources. Calculation details can be found in Tables C1–C6. Figure 8 shows the proportion of the assets in our sample for which the STTM index has significant Granger predictive power in each of the studied years.

Weekly trading strategy performance

In total, we compare the performance of 28 different portfolios: five endogenous model based portfolios, 20 portfolios using shallow feature based methods of text processing, two portfolios based on the proposed STTM approach, and one based on SESTM. Each of these mentioned portfolios is constructed for three different news sources independently. Given that we have 39 tickers in our analysis, in total we obtain 2,886 different models validated on six train/test splits each. We also compare all these models to two baselines: MOEX Russia Index, as a type of a broad stock market index, introduced above, and Equal Weight Index (EWI) based on MOEX’s tickers, as a type of buy and hold strategy. While the former exemplifies capitalization-weighted index, the latter gives equal weight to all stocks, including small-cap stocks that are generally considered to be higher risk and to have higher potential return investments compared to large-caps. In theory, giving greater weight to the smaller names of the MOEX Russia Index in an equal-weight portfolio should increase the return potential of the portfolio, so EWI may be expected to perform better than MOEX Russia Index.

Table 2 contains information about performance of the baselines. Table 3 demonstrates the results for topic modeling based approaches, Table 4 contains the results for shallow feature based methods of text processing and Table 5 presents the results for the endogenous models.

Figure 9 shows how weekly returns accumulate depending on the model used for forming an investment strategy and compares them to MOEX Russia Index (IMOEX) and Equal Weight Index (EWI) as baseline strategies. Each strategy uses one-week-ahead approach and sorts portfolios by the score obtained from the chosen model. Specifically, Figs. 9A–9C plot portfolio return accumulation graphs for the strategies based on one of the eight models (STTM (LDA), STTM (DTM), SESTM approaches and five best returns of strategies built on shallow feature based methods of text processing). Each facet A-C presents strategies using the data from only one information source: Kommersant, RIA Novosti and Vedomosti, respectively. Figure 9D contains the best models that have a Sharpe Ratio greater than one. We discuss it in detail further below.

Granger causality tests

Figure 8 shows that, as the time passes and the volume of the training data increases, the proportion of tickers for which our STTM approach turns to be Granger-causal tends to increase as well. An exception is a sharp fall of the predictive power for the majority of models in 2018. This fall is most probably explained with a number of international macroeconomic events in the second half of 2018 (including USA-China trade wars, US Federal Funds Rate hike, and the collapse of a large number of global financial indices). These events were poorly covered in the Russian media which focused on the internal agendas, such as the resonant raise of the retirement age. From Fig. 8 it can be seen that, according to the Granger causality test, the proposed STTM index can be significant for as much as 70% of stock quotes listed in the MOEX Russia Index if the data is sufficient to calibrate the model.

Weekly trading strategy performance

The D facet of Fig. 9 illustrates one-week-ahead performance of most economically successful portfolios (Sharpe ratio more than one), as well as the MOEX Russia Index. The top models in terms of the success of the investment portfolio built on them are as follows: STTM (LDA) on Kommersant with mean Sharpe ratio 1.3706 (annual return 36.12%), GBM + Navec based on RIA Novosti first paragraph with Sharpe ratio 1.1917 (annual return 32.84%), STTM (LDA) on Vedomosti with mean Sharpe ratio 1.1059 (annual return 28.40%), STTM (DTM) on Kommersant with mean Sharpe ratio 1.0798 (annual return 28.45%), STTM (LDA) on RIA Novosti with mean Sharpe ratio 1.0140 (annual return 27.55%), and LR + Doc2Vec based on Kommersant’s titles 1.0056 (annual return 27.46%). Portfolios built on the models mentioned above are also the most profitable. So out of 69 different approaches to stock trend prediction, only six turned out to be economically viable. Among them, four are built on our novel proposed approach STTM and only two are derived using shallow feature based text processing methods. Three out of six are based on Kommersant news agency data, two are based on RIA Novosti data, one is based on Vedomosti.

Now let us take a closer look at the best models for each individual news agency. A, B, C facets of Fig. 9 display performance for news sources Kommersant, RIA Novosti and Vedomosti, respectively. Each facet includes all topic modeling based approaches and five most successful models out of the remaining approaches (shallow feature based methods of text processing and endogenous models). For the Kommersant news agency, the STTM (LDA), model takes the first place (Sharpe ratio 1.3706 and annual return 36.12%), followed by STTM (DTM) (Sharpe ratio 1.0798 and annual return 28.45%) both in terms of Sharpe ratio and annual return. Next are five models with shallow feature based methods of text processing. For RIA Novosti STTM (LDA) and STTM (DTM) take the second (Sharpe ratio 1.0140 and annual return 27.55%) and the fourth (Sharpe ratio 0.9691 and annual return 26.75%) places respectively, the rest of the best models are shallow feature based methods of text processing. For Vedomosti STTM (LDA) and STTM (DTM) take the first (Sharpe ratio 1.1059 and annual return 28.40%) and third (Sharpe ratio 0.7941 and annual return 20.95%) places, respectively, the remaining best models again are shallow feature based methods of text processing.

Our experiments show that for all news sources the proposed STTM approach is among the best models, while maintaining the interpretability of results (see Figs. B9, B10, B11 and B12 in Appendix B). It is worth noting that endogenous models do not make it to the top of the best models, which in turn indicates that more useful economic information can be obtained from external data sources as compared to the information contained in the time series. SESTM never gets in any list of the best strategies, possibly, due to a smaller size of our dataset as compared to the dataset used by SESTM developers. However, we note that SESTM still outperforms the general economic baseline MOEX Russia Index both in terms of Sharpe ratio and annual return.