From words to returns: sentiment analysis of Japanese 10-K reports using advanced large language models

Computing infrastructure

We used Google Colaboratory, a cloud-based platform provided by Google. The computing environment is as follows.

– CPU: Intel Xeon 2.20 GHz.

– GPU: NVIDIA Tesla T4 GPU (RAM 15.0 GB).

– Operating System: Ubuntu 22.04.4 LTS (as provided by Google Colaboratory).

– Python Version: Python 3.11.

– Key Libraries: Detailed information on the libraries used is provided within the notebook.

Additionally, the code and datasets used in this study are publicly available at (https://doi.org/10.5281/zenodo.14892526) and are provided as supplemental files. The details of the hyperparameters are in ‘API Settings for Large Language Models’.

Data collection

To investigate the relationship between management sentiment and future stock returns in the Japanese market, we collected a comprehensive dataset of 10-K reports. The data spans a 10-year period, starting in 2014—the first year comprehensive reports were systematically available from Japan’s Electronic Disclosure for Investors’ NETwork (EDINET)—through 2023. From 2014 to 2022, our sample includes all firms listed on the former Tokyo Stock Exchange (TSE) First Section. Following a major market reform in April 2022, this section was replaced by the new Prime Market. Accordingly, for the 2023 fiscal year, our sample consists of firms listed on the Prime Market, which has stricter listing requirements. This change in the underlying sample universe explains the smaller number of firms in our final year. To ensure consistency in reporting cycles, our analysis focuses exclusively on companies with a fiscal year ending in March.

Data source

The primary source of the 10-K reports is the Electronic Disclosure for Investors’ NETwork (EDINET), an electronic disclosure system operated by Japan’s Financial Services Agency. EDINET provides access to official financial filings submitted by companies listed on Japanese stock exchanges, ensuring the authenticity and completeness of the data collected. We downloaded the annual financial reports for all firms listed on the TSE during the sample period, focusing exclusively on companies whose fiscal year ends in March, which is the case for most listed firms on the TSE (Please visit https://disclosure2.edinet-fsa.go.jp/WEEK0010.aspx for details.). This selection results in a total of 16,363 firm-year observations, encompassing over 90 million words of textual data. The size of this dataset provides a substantial basis for analyzing management sentiment across a broad cross-section of the Japanese market.

Table 1 summarizes the dataset, detailing the number of firms (10-K reports) per year, the total number of stocks analyzed, and the average length of the Management Discussion and Analysis (MD&A) sections in terms of the number of characters. The focus on the MD&A section is critical, as it provides management’s narrative on financial condition, operational results, and cash flow status, equivalent to the MD&A in U.S. filings. This section is particularly rich in qualitative information that can reflect management’s outlook and potentially influence investor perception.

Sentiment extraction methods¹

To extract sentiment from the Management Discussion and Analysis (MD&A) sections, we employed five different methodologies: a simple tone calculation using a financial polarity dictionary specifically developed and trained on Japanese financial texts, DeBERTaV2, and three advanced LLMs—GPT-4, Claude, and Gemini. The simple tone calculation involves two distinct metrics: the Tone Ratio, which measures the balance between positive and negative expressions by comparing the frequency of positive words to that of negative words, and the Tone Score, which captures the overall sentiment strength by summing up the individual sentiment scores assigned to each word in the text. By comparing these traditional dictionary-based methods, such as Tone Ratio and Tone Score, with state-of-the-art NLP models, we aim to assess the effectiveness of each approach in accurately capturing management sentiment and understanding its implications for future stock returns.

Simple tone calculation

To calculate the tone, we utilized the Financial Polarity Dictionary developed by the University of Tokyo (Visit https://sites.google.com/socsim.org/izumi-lab/tools/ for more information.). This lexicon is specifically designed for financial text analysis and provides two key pieces of information for each word, which we use to construct two distinct sentiment metrics: a Tone Ratio and a Tone Score. First, the dictionary categorizes each word as positive, negative, or neutral. This classification allows us to measure the balance of sentiment in a document. Following the approach of Li (2010), we compute the Tone Ratio as:

$$ToneRatio={\displaystyle \frac{N^{+}-N^{-}}{N^{+}+N^{-}}}$$where $N^{+}$ is the number of positive words and $N^{-}$ is the number of negative words in the document.

Second, the lexicon assigns a real-numbered score to each word, reflecting its degree of positivity or negativity. Words with strongly positive connotations are assigned higher positive scores, while words with strongly negative connotations receive negative scores. Neutral words are assigned a score of zero. Using these fine-grained scores, we calculate the Tone Score as the cumulative sum of all individual word scores in the Management Discussion and Analysis (MD&A) section:

$$ToneScore=\sum _{i=1}^n{s}_i{s}_i\in \mathbb{R}$$where $s_i$ is a real number expressing the sentiment score of the $i$-th word and the sum is taken over all $n$ words in the document. This provides a different measure of overall sentiment compared to the proportional Tone Ratio, which is normalized by the total count of sentimental words rather than the total document length.

DeBERTaV2 model

For our transformer-based analysis, we employ a fine-tuned DeBERTaV2 model. We selected this architecture due to its documented performance improvements over earlier models, particularly its enhanced disentangled attention mechanism which is effective at capturing the nuances of complex sentences common in financial disclosures. Our implementation began with a Japanese-language DeBERTaV2 base model, pre-trained on a large general corpus. We then fine-tuned this base model specifically for financial sentiment classification using the chABSA² dataset, a labeled corpus of sentences from Japanese financial reports. This specialization process adapts the model to the specific vocabulary and contextual phrasing of our target domain.

Let $X$ represent the input text sequence tokenized into subword units. The MLM objective is optimized by masking a fraction of tokens in $X$, and the loss is computed as:

$$L_{MLM}=-{\displaystyle \frac{1}{\left|M\right|}\sum _{i\in M}logP(x_i|X_{\backslash i})}$$where $M$ is the set of masked token positions, $X_{\backslash i}$ represents $X$ with the i-th token masked, and $P(x_i|X_{\backslash i})$ is the predicted probability of the original token $x_i$. For the sentiment classification task, the model predicts a probability distribution over sentiment class $C=\{positive,negative,neutral\}$ for each sentence $X$. The sentiment classification loss is:

$$L_{SC}=-{\displaystyle \frac{1}{N}\sum _{j=1}^N\sum _{c\in C}{y}_{j,c}logP(c|X_j)}$$where N is the number of training samples and $y_{j,c}\in \{0,$1} is the ground-truth label for class $c$ for the $j$-th sentence. $P(c|X_j)$ is the predicted probability for class $c$. The fine-tuning combines both losses:

$$L=L_{MLM}+\lambda L_{SC}$$where $\lambda$ is the hyperparameter.

The model is fine-tuned using the chABSA dataset, which consists of sentences extracted from Japanese financial reports with sentiment labels assigned to specific terms. Each sentence is labeled with its sentiment polarity, indicating whether the expressed sentiment is positive, negative, or neutral. To ensure focus on unambiguous sentiment, we select sentences where all sentiment annotations are uniformly positive or negative, resulting in a subset of n = 2,227 sentences. The DeBERTaV2 model processes each sentence $X_j$ independently, generating a probability distribution $P(c|X_j)$ over the sentiment classes $c\in C$. For each document $D$ consisting of $m$ sentences, the final sentiment score $S\left(D\right)$ is computed as a weighted average:

$$S\left(D\right)=\sum _{j=1}^m{w}_j\cdot \sum _{c\in C}s\left(c\right)\cdot P(c|X_j)$$where $w_j$ is the weight assigned to the $j$-th sentence, determined by the model’s confidence and $s\left(c\right)$ is the numerical score assigned to each sentiment class.

The DeBERTaV2 model processes each sentence in the MD&A sections independently, generating a probability distribution over the three sentiment classes. The final sentiment score for each document is computed as a weighted average of these probabilities, with weights determined by the model’s confidence in each prediction.

GPT-4

GPT-4 is a state-of-the-art language model developed with deep learning techniques that enable it to comprehend and generate human-like text across a variety of domains. The model demonstrates remarkable capabilities in understanding nuanced language, making logical connections, and adapting its response style based on specific task requirements. A key strength of GPT-4 is its ability to process and synthesize information from multiple perspectives, making it particularly effective for tasks requiring both analytical depth and contextual awareness.

In our study, we specifically utilize GPT-4o-mini (gpt-4o-mini-2024-07-18), which is optimized for efficient processing while maintaining high accuracy in sentiment analysis tasks. To guide GPT-4 in generating consistent and accurate sentiment scores, we crafted a detailed prompt in Japanese tailored to the financial context of the MD&A sections.

The prompt instructs the model to act as a securities analyst, read the company’s disclosure information, and calculate positive and negative sentiment scores that sum to 100, considering the context of the entire document. It includes specific rules and examples to ensure that the model’s output aligns with our analytical objectives. For instance, the model is instructed to output only the positive and negative scores without providing reasons or additional commentary, following a strict format for consistency.

An excerpt of the prompt is as follows (translated for clarity):

• “You are a securities analyst. Read the company’s disclosure information below and, considering the context, calculate the positive sentiment score and negative sentiment score so that they sum up to 100.

• Rules:

• – Output in the following format: positive score: X (0–100), negative score: Y (0–100).

• – Ensure that the positive score and negative score add up to 100.

• – Evaluate the entire content of the disclosure information thoroughly and precisely, referencing the examples.

• – Do not output reasons; only output positive score and negative score.”

By providing explicit instructions and examples of fully positive and fully negative disclosures, the prompt ensures that GPT-4 applies a consistent evaluation framework across all MD&A sections. The prompt detail is in the ‘LLM Hyperparameters and Prompt Details’.

Claude

Claude employs a unique approach to language modeling that sets it apart from both traditional dictionary-based methods and other large language models. While GPT-4 and Gemini rely on carefully designed prompts to guide their sentiment evaluations, Claude’s architecture emphasizes precise instruction-following and a deep, context-sensitive understanding of the text. Claude’s core strength lies in its ability to maintain logical consistency throughout complex analyses and consider multiple angles simultaneously. Whereas other LLMs may be more heavily influenced by their training data or specific examples provided in the prompt, Claude integrates instructions and contextual cues to produce balanced, well-reasoned sentiment assessments.

For our analysis, we employ Claude 3 Haiku (claude-3-haiku-20240307), configured to provide efficient processing while maintaining high accuracy in sentiment evaluation. Like GPT-4, we designed a standardized prompt for Claude in Japanese, instructing the model to produce positive and negative sentiment scores summing to 100 for each MD&A section. This approach ensures compatibility with our other analysis methods while leveraging Claude’s sophisticated natural language understanding capabilities to provide nuanced sentiment assessments.

Gemini

Gemini, developed by Google, distinguishes itself from models like GPT-4 and Claude in how it arrives at sentiment judgments without relying as heavily on explicit instructions or prompt engineering. While GPT-4 and Claude are notably effective at following detailed guidance—often producing sentiment assessments that closely align with the format and criteria specified in their prompts—Gemini’s approach is less about adherence to predetermined rules and more about organically interpreting the text’s underlying context. Unlike GPT-4, which excels at reasoning through instructions and examples to generate consistent output, and Claude, which emphasizes precise instruction-following and logical consistency, Gemini focuses on weaving together multiple layers of context—both at the micro-level of individual words and the macro-level of the document’s overarching narrative. This means that instead of primarily depending on how well the prompt guides it, Gemini dynamically recalibrates its understanding as it processes the text, extracting nuanced sentiment shifts that may emerge without direct cues or explicit scoring rules.

In our implementation, we utilize Gemini 1.5 Flash (gemini-1.5-flash), applying the same standardized scoring approach used with our other LLM implementations. The model evaluates each MD&A section to produce positive and negative sentiment scores that sum to 100, ensuring consistency across all analyses. The model processes the documents using a context-aware approach that considers both local semantic relationships and broader thematic elements in determining sentiment scores.

Incorporating custom prompts in LLMs

A critical aspect of leveraging LLMs for sentiment analysis in financial documents lies in the careful design of input prompts. In this study, we have employed carefully structured prompts to guide models such as GPT-4, Claude, and Gemini, ensuring that their responses align with the analytical goals of the research. For GPT-4, a tailored prompt was developed in Japanese, directing the model to adopt the perspective of a securities analyst when evaluating the disclosed material. This prompt included explicit instructions on how to interpret and report sentiment, including detailed guidelines on scoring and a range of examples illustrating both strongly positive and distinctly negative cases. By providing the model with a clear evaluative framework and concrete textual references, it becomes possible to produce highly consistent sentiment measures across multiple documents. This approach also ensures that the model focuses on the essential aspects of the disclosures rather than drifting into irrelevant commentary. It is important to emphasize that no alterations were made to the original MD&A language. Working directly with the source text ensures that the derived sentiment scores accurately capture the tone and content that the management intended to convey, thereby producing results that are well-positioned to inform subsequent analyses of the relationship between sentiment and financial outcomes.

Sentiment based long-short portfolio return

Table 2 presents the summary statistics for the sentiment scores derived from our six different methods. A notable pattern emerges from these distributions: the mean scores for the dictionary-based methods (Tone Ratio and Tone Score) are negative, while those for the LLMs are positive. This is consistent with prior literature (e.g., Loughran & McDonald, 2011), which finds that finance-specific dictionaries tend to have a negative skew, as corporate disclosures often employ cautious language to mitigate legal risk. In contrast, our LLM scores are derived from a prompt that instructs the models to allocate scores summing to 100 between positive and negative sentiment, naturally centering the distribution around a positive mean (0.5 after normalization). We now proceed to test the predictive power of these different sentiment measures for future stock returns.

The primary aim of this study is to investigate whether large language models (LLMs) can identify information within Japanese 10-K reports that anticipates future stock returns. Under the Efficient Market Hypothesis (EMH) outlined by Fama (1970), all public information should already be embedded in asset prices, rendering attempts to achieve abnormal returns through conventional analysis futile. However, previous empirical work points to a more complex reality. Studies such as Tetlock (2007) and Loughran & McDonald (2011) reveal that textual characteristics of corporate disclosures can encode signals predictive of subsequent stock performance. This tension between theory and empirical evidence raises the question of how novel analytical tools—particularly LLMs—might bridge the gap.

To explore the predictive value of such sentiment measures, this research constructs value-weighted portfolios sorted on sentiment extracted from the Management Discussion and Analysis (MD&A) sections of 10-K reports. For each fiscal year from 2014 through 2023, all firms listed on the Tokyo Stock Exchange are ranked according to their sentiment scores as of June. These scores, derived from the previous fiscal year’s filings ending in March, determine which firms enter the long and short portfolios. The top quintile of firms with the highest sentiment forms the long portfolio, and the bottom quintile with the lowest sentiment forms the short portfolio. Both portfolios are value-weighted by market capitalization at the formation date and are held for one year, spanning from July of year t to June of year t + 1. We select value-weighting for our primary analysis as this approach reflects the performance of a realistic, investable portfolio and is standard in the asset pricing literature. A corresponding analysis using equal-weighted portfolios is presented in the “Robustness Check” section. This procedure repeats annually for each of the six sentiment extraction methods—Tone Ratio, Tone Score, DeBERTaV2, GPT-4, Claude, and Gemini—providing a comparative framework that evaluates conventional and advanced approaches in tandem.

Figure 2 presents the cumulative returns for the long-short portfolios derived from each method. The results reveal important distinctions in how these methods capture return-predictive sentiment. Contrary to conventional expectations, the long-short portfolios constructed with GPT-4 and Claude sentiment measures exhibit sustained negative cumulative returns. This finding indicates that firms displaying high positive sentiment underperform those classified as having strong negative sentiment, suggesting that positive sentiment may be overvalued, negative sentiment may be overvalued, or both. The portfolios based on Tone Ratio and Tone Score, while grounded in dictionary-based approaches, show only modest or even negative results, indicating that these traditional metrics do not isolate the type of sentiment that correlates with future stock performance. DeBERTaV2 similarly struggles to produce strong predictive signals. The Gemini-based portfolios produce intermediate results, performing better than the traditional methods yet not reaching the levels of GPT-4 and Claude. Together, these findings imply that certain LLM architectures yield a more nuanced sentiment measure that aligns with future returns, challenging the assumption that all publicly available information is already reflected in share prices.

To decompose the result from Fig. 2 and provide a more granular view from an investor’s perspective, Fig. 3 presents the cumulative annual buy-and-hold returns for the long and short legs of the portfolios separately, benchmarked against the TOPIX index. The decomposition clearly illustrates the source of the contrarian signal. Panel A shows that the portfolio composed of the highest-sentiment stocks consistently underperforms the TOPIX benchmark. In contrast, Panel B demonstrates that the portfolio of the lowest-sentiment stocks significantly outperforms the benchmark. This clarifies why the long-short strategy in Fig. 2 generated negative returns: the long leg of the strategy (the positive-sentiment portfolio in Panel A) underperformed, while the assets in the short leg (the negative-sentiment portfolio in Panel B) outperformed the benchmark, leading to losses on the short positions.

Asset pricing test

To evaluate the performance of sentiment-based portfolios, we employ standard asset pricing models that describe stock returns as functions of systematic risk factors. These models serve to determine whether the returns of a portfolio can be explained by exposure to well-known risk factors or whether they demonstrate abnormal performance, often referred to as alpha.

The analysis begins with the Fama-French three-factor model (FF3), which explains portfolio returns based on three primary components: the market excess return, the size premium, and the value premium (Fama & French, 1993). The size premium captures the historical tendency of smaller firms to outperform larger firms, while the value premium reflects the superior returns of firms with high book-to-market ratios relative to those with low ratios. The model is expressed as:

$$R_{pt}-R_{ft}={\alpha }_p+{\beta }_1\left(R_{Mt}-R_{ft}\right)+{\beta }_2\left({\lambda }_{SMB,t}\right)+{\beta }_3\left({\lambda }_{HML,t}\right)+{\epsilon }_{pt}$$where $R_{pt}$ is the return of portfolio at time t, $R_{ft}$ is the risk-free rate, $R_{Mt}-R_{ft}$ represents the market excess return, ${\lambda }_{SMB,t}$ denotes the size premium, and ${\lambda }_{HML,t}$ represents the value premium. The intercept αp captures the portfolio’s abnormal return, and ${\epsilon }_{pt}$ is the idiosyncratic error term.

Next, we extend the analysis by incorporating the momentum factor, which accounts for the observed tendency of stocks with strong past performance to continue outperforming those with weak past performance. This results in the Carhart four-factor model (FFC4), which is given by (Please see Carhart (1997)):

$$R_{pt}-R_{ft}={\alpha }_p+{\beta }_1\left(R_{Mt}-R_{ft}\right)+{\beta }_2\left({\lambda }_{SMB,t}\right)+{\beta }_3\left({\lambda }_{HML,t}\right)+{\beta }_4\left({\lambda }_{WML,t}\right)+{\epsilon }_{pt}.$$

In this equation, ${\lambda }_{WML,t}$ captures the momentum premium, reflecting the returns of winning stocks relative to losing stocks.

Finally, we consider the Fama-French five-factor model (FF5), which introduces two additional factors to account for profitability and investment patterns (Fama & French, 2015). Profitability captures the distinction between robust firms and weaker ones, while the investment factor contrasts conservative firms with aggressive ones in terms of capital expenditure. The FF5 model is expressed as:

$$R_{pt}-R_{ft}={\alpha }_p+{\beta }_1\left(R_{Mt}-R_{ft}\right)+{\beta }_2\left({\lambda }_{SMB,t}\right)+{\beta }_3\left({\lambda }_{HML,t}\right)+{\beta }_4\left({\lambda }_{RMW,t}\right)+{\beta }_5\left({\lambda }_{CMA,t}\right)+{\epsilon }_{pt}$$

Here, ${\lambda }_{RMW,t}$ represents the profitability premium, and ${\lambda }_{CMA,t}$ denotes the investment premium.

These models allow us to test whether the performance of sentiment-based portfolios can be attributed to common risk factors or if they exhibit abnormal returns. Specifically, a statistically significant intercept ${\alpha }_p$ indicates the presence of abnormal performance that cannot be explained by the risk factors included in the model.

Table 3 reports the estimated alphas and factor loadings from these regressions for each portfolio. The results indicate that the long-short portfolios based on GPT-4 and Claude sentiment scores yield statistically significant negative alphas across all models. Specifically, the GPT-4-based portfolio exhibits annualized alphas of −5.95% (FF3, t-statistic = −2.01), −6.49% (FFC4, t-statistic = −2.21), and −6.29% (FF5, t-statistic = −2.18), all significant at the 5% level. Similarly, the Claude-based portfolio shows even more pronounced negative alphas of −9.15% (FF3, t-statistic = −2.66), −9.90% (FFC4, t-statistic = −2.92), and −9.46% (FF5, t-statistic = −2.76), all significant at the 1% level. Beyond their statistical significance, the abnormal returns generated by the LLM-based strategies are also of considerable economic magnitude. To place our findings in context, the annualized alpha for the GPT-4 portfolio was −5.95% and −9.15% for the Claude portfolio under the Fama-French three-factor model. During our sample period, the average annualized return for well-established factors in the Japanese market, such as the size premium (SMB), was 1.39%, while the value premium (HML) was approximately 1.35%. The magnitude of the negative alpha captured by our sentiment strategy is therefore exceeds the returns from these foundational market anomalies. This comparison highlights that the predictive signals extracted by sophisticated LLMs are not just statistically robust but also represent an economically significant market inefficiency.

These results suggest that sentiment-based strategies utilizing GPT-4 and Claude systematically underperform relative to expectations based on common risk factors. In contrast, other sentiment-based methods, such as the Tone Ratio, Tone Score, DeBERTaV2, and Gemini, produce alphas that are not statistically significant, indicating that these portfolios fail to generate abnormal returns after accounting for conventional risk factors. These findings challenge the weak form of the Efficient Market Hypothesis (EMH) by demonstrating that advanced large language models (LLMs) can extract subtle but predictive information from publicly available 10-K reports that is not fully reflected in stock prices at the time of disclosure.

To place our findings within the context of the broader literature, it is useful to compare our results with other studies that use LLMs for return prediction. Our finding of a significant negative alpha (e.g., −5.95% annualized for the GPT-4 portfolio under the FF3 model) presents a notable contrast to some recent work. For example, Lopez-Lira & Tang (2023), as cited in the survey by Nie et al. (2024), found that ChatGPT could forecast stock movements with a positive correlation when analyzing news headlines. This divergence suggests that the source and nature of the text are critical. The formal, backward-looking, and carefully curated language of a 10-K report may be processed by investors as a contrarian signal—where overly positive sentiment suggests overvaluation—whereas the immediate and forward-looking nature of news headlines may serve as a more direct momentum signal. Our results align more closely with the behavioral finance theory of investor overreaction to corporate disclosures, which is then corrected over time.

A key finding of this study is that sentiment scores derived from advanced LLMs have significant power to predict future stock returns, a result that contributes to a rapidly advancing body of research on the application of LLMs in finance. A recent survey by Nie et al. (2024) highlights a study demonstrating that an LLM, when given only anonymized, numerical financial statements, can outperform professional human analysts in predicting the direction of future earnings. The survey notes that the LLM succeeds not from memory, but by generating its own analytical insights from the raw financial data. Our research complements and extends this finding by tackling the challenge from the opposite direction. Whereas some studies highlighted by Nie et al. (2024) deliberately exclude narrative text to isolate the LLM’s numerical reasoning ability, our study focuses specifically on extracting sentiment from the narrative-rich Management Discussion and Analysis (MD&A) section. Their work showed that LLMs can reason effectively from quantitative data alone; our work shows they can extract predictive signals from the qualitative discussion that accompanies it. These parallel findings suggest that advanced LLMs possess a comprehensive capability to generate alpha from both the quantitative and qualitative components of corporate financial disclosures.

This linguistic capability leads to a second key insight rooted in behavioral finance: investor overreaction. When LLMs identify high sentiment, they are effectively flagging firms whose optimistic disclosures are persuasive enough to cause an initial overreaction from investors, driving stock prices above their fundamental value. The subsequent negative abnormal returns, which we document as a statistically significant alpha, can be interpreted as the market’s eventual correction of this initial overpricing. This interpretation finds further support when considered in the context of the well-known “momentum puzzle”. The momentum effect—a globally observed anomaly where stocks with high past returns (“winners”) continue to outperform those with low past returns (“losers”)—has been shown to be notably absent in the Japanese market (Jegadeesh & Titman, 2023). Our long-short portfolio exhibits a positive loading on the momentum factor (WML), indicating that firms with high LLM-derived sentiment tend to be past winners. However, the fact that these portfolios subsequently generate negative alpha suggests a market structure where the overreaction to positive signals leads to a strong reversal, rather than the sustained performance characteristic of momentum. Thus, our findings align with a market environment that does not reward momentum and, in this context, punishes apparent ‘winners’ identified through textual sentiment.

Case study: analysis of portfolio outliers

To provide a more granular analysis of our portfolio-level findings, Table 4 presents a case study of stocks with extreme performance outcomes.

The analysis is divided by the model used for the initial classification: Panel A is based on GPT-4’s sentiment scores, and Panel B is based on Claude’s. For the “Best Performers” in each panel, we first identified all firms that the anchor model classified as having negative sentiment and then selected the top three performers based on their subsequent 1-year buy-and-hold returns (BHR). Conversely, the “Worst Performers” are the three firms with the lowest BHR, selected from the pool of firms classified as having positive sentiment. The table displays each stock’s sentiment rank from both GPT-4 and Claude. Since ranks are based on the sentiment scores generated by the LLMs, multiple firms can share the same rank each year if their underlying scores are identical; for example, all three “Worst Performers” under the GPT-4 classification in 2021 share the same rank of 44. The table reveals a high degree of agreement between the models on these extreme cases. Notably, when a stock’s MD&A was categorized as positive by GPT-4, Claude’s sentiment rank was also strongly positive for the most part. This cross-model consensus on the outlier classifications provides further support for the validity of the extracted sentiment signal. These cases provide concrete examples of the study’s central contrarian finding that a negative sentiment classification can precede exceptionally strong returns, while a positive classification can precede significant underperformance.

Alternative portfolio formation thresholds

Our primary analysis uses the top and bottom 20% of firms to form portfolios. To ensure our conclusions are not sensitive to this specific threshold, we reconstruct the portfolios using both the top/bottom 10% and top/bottom 30% of firms based on sentiment scores. The results, summarized in Table 5, remain consistent with our main analysis, confirming that our findings are robust to the choice of portfolio formation threshold.

Portfolio weighting scheme

Our main analysis employs value-weighted portfolios to reflect a realistic investment strategy. To test the sensitivity of our findings to this choice, we conduct a robustness check using equal-weighted long-short portfolios, with the results presented in Table 6. When using an equal-weighted scheme on the full sample of firms (Panel A), the alpha for the GPT-4 portfolio becomes statistically insignificant, whereas the Claude portfolio retains weak to moderate statistical significance. This mixed result suggests a complex interaction between the sentiment signals and firm size, motivating a more direct test on large-capitalization stocks.

We investigate this further by focusing on a subsample of large-cap firms: the constituents of the TOPIX100 index (Panel B). In this test, the results diverge dramatically. The alpha for the GPT-4 portfolio now becomes strongly and consistently significant across all asset pricing models. In contrast, the alpha for the Claude portfolio becomes insignificant. The disappearance of the Claude portfolio’s alpha can be explained by its construction. We find that Claude’s sentiment scores result in a highly concentrated long portfolio, with an average of only 8.6 stocks over 10 years. Gemini and GPT sentiment scores, on the other hand, generate more than double the number of stocks. While its signal is detectable in the larger full sample, this concentration makes the portfolio’s return highly sensitive to firm-specific risk, which likely obscures the alpha signal in the equal-weighted TOPIX100 test. This heterogeneity is a key finding, confirming that the sentiment effect is essentially a phenomenon of large-cap stocks and demonstrating that different LLMs capture signals with distinct characteristics.

Adjustment for transaction costs

To assess the real-world implementability of our strategy, we model an investor seeking to exploit the documented anomaly by shorting top-quintile (high-sentiment) stocks and buying bottom-quintile (low-sentiment) stocks. We account for trading frictions by incorporating a conservative 1.5% round-trip transaction cost, an assumption grounded in the empirical findings of Frazzini, Israel & Moskowitz (2018). To embed this cost, we adjust the transaction prices to reflect a “buy high, sell low” scenario: for long positions, we assume the investor buys at a 0.75% higher price and sells at a 0.75% lower price 1 year later. Conversely, for short positions, the investor sells at a 0.75% lower price and buys back at a 0.75% higher price. Since our primary asset pricing tests (Table 3) showed that the strategies based on Tone Ratio, Tone Score, and DeBERTaV2 did not generate significant alpha, we focus this post-cost analysis exclusively on the three LLM-based strategies. The detailed yearly performance for these strategies, net of costs, is presented in Table 7.

The table breaks down the annual buy-and-hold returns by long-only, short-only, and combined long-short portfolios for each LLM-based strategy. The results show that even after accounting for significant trading frictions, the strategies based on GPT-4 and Claude sentiment remain profitable over the whole sample period, generating cumulative returns of 34.0% and 58.1%, respectively. This confirms that the predictive power of the advanced LLMs is not only statistically significant but also robust enough to represent a viable trading strategy under realistic cost assumptions.

Exclusion of the 2023 sample

To ensure our results are not driven by the change in the sample universe following the 2022 TSE market reform, we perform an additional robustness check. We re-run our main asset pricing tests on a sample that excludes the 2023 data, using only the consistent TSE First Section sample from 2014–2022. We confirm that all our main findings, including the statistically significant negative alphas for the GPT-4 and Claude portfolios, remain qualitatively unchanged. This indicates that our conclusions are robust and not contingent on the inclusion of the final year’s Prime Market data.

API settings for large language models

To ensure the reproducibility of our study, this section details the specific models and API hyperparameter settings used for sentiment extraction. It is important to note that our prompt strictly instructs the models to return only two numbers (a positive and a negative score) and to refrain from generating any narrative text. Consequently, hyperparameters that primarily control the creativity and linguistic diversity of generated sentences, such as presence_penalty and frequency_penalty, are largely irrelevant to our specific task. However, we specified parameters like a low temperature and top_p across all models to ensure that the numerical output is as deterministic and consistent as possible, which is crucial for a research context. Table A1 summarizes the settings employed for GPT-4o-mini, Claude 3 Haiku, and Gemini 1.5 Flash.

Prompt for LLM sentiment scoring

The same standardized prompt was used for GPT-4o-mini, Claude 3 Haiku, and Gemini 1.5 Flash to generate sentiment scores. The prompt was provided to the models in Japanese to analyze the original Japanese 10-K report texts. The original prompt is presented below, followed by an English translation for clarity.

Original Japanese prompt

あなたは証券アナリストです.以下の企業の開示情報を読み、文脈も考慮して、positive_sentiment_scoreとnegative_sentiment_scoreを合計100になるように算出しなさい˚

ルール)

フォーマットは以下で出力しなさい.positive_score: X (0–100), negative_score: Y (0–100)

positive_scoreとnegative_scoreは、両方合わせて100になるようにスコアを算出しなさい˚

例を参考にして開示情報の内容全体を漏れなく厳密に評価しなさい˚

理由は出力しないで、positive_score: X (0–100), negative_score: Y (0–100)のみを出力してください˚

• positive_scoreが100の開示情報の例

「セグメント利益は、販売数量の増加により、同247百万円増益の178百万円となりました」

「米国では企業業績や個人消費が堅調に推移し、景気は緩やかに回復しました」

「営業利益は、非鉄金属相場や為替相場の変動に伴うたな卸資産の在庫影響(以下「在庫要因」)が好転し、機能材料部門において主要製品の販売量が増加したこと等により、前連結会計年度に比べて273億円(245.3%)増加の384億円となりました」

「期間限定で東京駅一番街にオープンした当社初のアンテナショップ「パティスリーブルボン」では、特別に仕立てたクッキーの限定商品「ラングレイス」や「ルマンドアソート」などに大きな反響をいただきました」

「セグメント利益は、のれん償却費３億 33 百万円を計上したものの、上記要因に伴う営業利益の増加に加え、飲食用資材分野における原材料価格の下落などにより、９億45百万円と前年同期比２億41百万円(34.3%)の増益となりました」

• negative_scoreが100の開示情報の例

「建築用塗料を取扱う塗料部門におきましては、新築向け市場及びリフォーム向け市場とも、工事を伴う施工棟数が前年度に比べ伸び悩んだことなどにより、売上高は減少いたしました」

「この結果、売上高は126億17百万円(同4.8％減)となり、営業利益は７億40百万円(同11.2％減)となりました」

「即席麺部門は、製造ラインの移設に伴う稼働率の低下と受託が低調に推移し、また、３月に製造ラインを増設しましたが、売上の寄与は低く、売上高は7,085百万円と前年同期と比べ659百万円(8.5%)の減収となり、セグメント利益(営業利益)は204百万円と前年同期と比べ219百万円(51.8%)の減益となりました」

「しかしながら、資材費や労務費のコストが高止まりする中で、北海道  ▪ 東北地区の集中豪雨の影響により、公共工事の優先順位が入れ替わり、当初予定されていた工期が先延ばしになるなど、当社を取り巻く経営環境は厳しい状況で推移しました」

「当事業年度におけるわが国経済は、政府による経済政策や金融政策の総動員もあり、緩やかな回復基調となったものの、個人消費や設備投資は力強さを欠き、海外経済の減速と為替、原材料価格の変動リスクを抱え、先行き不透明な状況が続いた」

English translation

You are a securities analyst. Please read the following corporate disclosure and, considering the context, calculate a positive sentiment score and a negative sentiment score so that they sum to 100.

Rules

Output in the following format: positive_score: X (0–100), negative_score: Y (0–100)

Ensure that positive_score and negative_score sum to 100.

Evaluate the entire content of the disclosure information thoroughly and precisely, referencing the examples.

Do not output any reasoning; only output the scores in the format positive_score: X (0–100), negative_score: Y (0–100).

Examples of disclosures with positive_score: 100

“Segment profit increased by 247 million yen over the same period last year to 178 million yen due to an increase in sales volume.”

“In the United States, corporate earnings and personal consumption remained strong, and the economy recovered moderately.”

“Operating income increased by 27.3 billion yen (245.3%) from the previous fiscal year to 38.4 billion yen, due to factors such as a favorable turn in inventory impact from non-ferrous metal and foreign exchange market fluctuations, and an increase in sales volume of major products in the functional materials division.”

“Our first pop-up shop, ‘Pâtisserie Bourbon,’ opened for a limited time at Tokyo Station’s First Avenue, received a great response for its specially crafted limited-edition cookies ‘Langlaze’ and ‘Lumonde Assortment’.”

“Although goodwill amortization of 333 million yen was recorded, segment profit increased by 241-million-yen (34.3%) year-on-year to 945 million yen, due to the increase in operating income from the factors above, as well as a decline in raw material prices in the food and beverage materials sector.”

Examples of disclosures with negative_score: 100.

“In the paint division, which handles architectural coatings, net sales decreased due to sluggish growth in the number of construction projects for both the new construction and renovation markets compared to the previous fiscal year.”

“As a result, net sales decreased to 12,617 million yen (a 4.8% decrease from the same period last year), and operating income decreased to 740 million yen (an 11.2% decrease from the same period last year).”

“The instant noodle division experienced a decrease in revenue of 659 million yen (8.5%) year-on-year to 7,085 million yen, and segment profit (operating income) decreased by 219 million yen (51.8%) to 204 million yen, due to a decline in operating rates from the relocation of production lines and sluggish contract manufacturing. Additionally, a new production line was added in March, but its contribution to sales was low.”

“However, while material and labor costs remained high, the business environment surrounding our company remained challenging, as the priority of public works projects shifted due to the impact of torrential rains in the Hokkaido and Tohoku regions, leading to the postponement of originally scheduled construction periods.”

“During this fiscal year, Japan’s economy showed a moderate recovery trend, partly due to the full mobilization of government economic and financial policies. However, personal consumption and capital investment lacked strength. With the risks of a slowdown in overseas economies and fluctuations in exchange rates and raw material prices, the outlook remained uncertain.”