Paper: arXiv 2411.00420

Authors: Kei Nakagawa, Masanori Hirano, Yugo Fujimoto

Abstract

This study aims to evaluate the sentiment of financial texts using large language models~(LLMs) and to empirically determine whether LLMs exhibit company-specific biases in sentiment analysis. Specifically, we examine the impact of general knowledge about firms on the sentiment measurement of texts by LLMs. Firstly, we compare the sentiment scores of financial texts by LLMs when the company name is explicitly included in the prompt versus when it is not. We define and quantify company-specific bias as the difference between these scores. Next, we construct an economic model to theoretically evaluate the impact of sentiment bias on investor behavior. This model helps us understand how biased LLM investments, when widespread, can distort stock prices. This implies the potential impact on stock prices if investments driven by biased LLMs become dominant in the future. Finally, we conduct an empirical analysis using Japanese financial text data to examine the relationship between firm-specific sentiment bias, corporate characteristics, and stock performance.

Complexity vs Empirical Score

  • Math Complexity: 3.5/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper employs a straightforward difference-in-means approach to quantify bias and uses a basic economic equilibrium model, indicating low-to-moderate mathematical complexity. However, it demonstrates high empirical rigor through the use of real-world Japanese financial text data, systematic testing across multiple LLMs, and an analysis linking sentiment bias to actual stock performance.

Research Flowchart

  flowchart TD
  Goal["Research Goal:<br>Evaluate LLM Sentiment Bias<br>in Financial Texts"]

  Method1["Method 1: Prompt Comparison<br>Score Texts With vs Without<br>Company Name"]
  Bias_Def["Define Bias = Score Difference"]
  Model["Method 2: Economic Model<br>Simulate Investor Behavior"]
  Data["Data: Japanese Financial<br>Texts & Stock Data"]

  Comp["Analysis: Compare Bias<br>vs Firm Characteristics &<br>Stock Performance"]

  Out1["Outcome: Quantified<br>Company-specific Bias"]
  Out2["Outcome: Theoretical Impact<br>on Stock Prices"]
  Out3["Outcome: Empirical Link<br>between Bias, Firms &<br>Performance"]

  Goal --> Method1
  Method1 --> Bias_Def
  Method1 --> Model
  Goal --> Data
  Data --> Comp
  Bias_Def --> Comp
  Model --> Out2
  Comp --> Out3
  Bias_Def --> Out1

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes to evaluate financial text sentiment with large language models, quantify company-specific bias, and build an economic model to assess its impact on investor behavior and stock prices.

Source sentence
We compare the sentiment scores of financial texts by LLMs when the company name is explicitly included in the prompt versus when it is not. We define and quantify company-specific bias as the difference between these scores. Next, we construct an economic model to theoretically evaluate the impact of sentiment bias on

Assumes 2

  • The market consists of both biased and unbiased investors who trade based on sentiment. market
    Source sentence
    Our model assumes a market where both biased and unbiased investors coexist, aiming to analyze how bias affects stock prices at equilibrium in such a market.
  • Unbiased investors consider sentiment in decisions, while biased investors trade based on sentiment bias. model
    Source sentence
    The unbiased investors consider sentiment in their investment decisions, while the biased investors trade based on sentiment bias, using biased sentiment in their decisions.

Data

  • Japanese Summary of Consolidated Financial and Business Results · Timely Disclosure network (TDnet) of the Tokyo Stock Exchang text-news public
    January 2019 to December 2023 · companies listed on the Tokyo Stock Exchange
    Source sentence
    We obtained the data from the Timely Disclosure network (TDnet1), a service provided by the Tokyo Stock Exchange. The data include Summary of Financial Results published between January 2019 and December 2023 by companies listed on the Tokyo Stock Exchange.

Costs

Not applicable The paper reports no trading or execution result.

Finds 3

  • Company-specific sentiment bias is present across multiple LLMs, with GPT-3.5 showing a negative evaluation of smaller firms when the company name is included. theoretical
    Source sentence
    This suggests that GPT-3.5 tends to negatively evaluate smaller companies when the company name is included in the prompt.
  • Theorem 1 provides an equilibrium price formula showing how sentiment bias shifts stock prices. theoretical
    Source sentence
    Theorem 1 (Impact of Bias on Equilibrium Price). Under the market conditions described above, the equilibrium price p* t is given by: ...
  • CARs over a 60‑day window differ for positively and negatively biased groups, indicating bias can affect abnormal returns. performance
    Source sentence
    Fig. 2,3, and 4 show the transition of the positive group, negative group and positive - Negative group CARs over a 60-day period for various LLMs, respectively.

Checked by others

Supported by

  • Your AI, Not Your View: The Bias of LLMs in Investment Analysis (2025)
    On the question: Do large language models show a company‑size sentiment bias in financial analysis?
    Compare the findings

    This paper: Company-specific sentiment bias is present across multiple LLMs, with GPT-3.5 showing a negative evaluation of smaller firms when the company name is included.

    That paper: Size bias scores decline as company size decreases, with statistically significant gaps between largest and smallest market‑cap quantiles.

    Both find that bias grows with size: GPT‑3.5 rates smaller firms more negatively, and bias scores drop as market‑cap decreases, with significant gaps between the largest and smallest quantiles.

Cited by 3 papers in the archive (3 by independent authors)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.