Evaluating Company-specific Biases in Financial Sentiment Analysis using Large Language Models

ArXiv ID: 2411.00420 “View on arXiv”

Authors: Unknown

Abstract

This study aims to evaluate the sentiment of financial texts using large language models~(LLMs) and to empirically determine whether LLMs exhibit company-specific biases in sentiment analysis. Specifically, we examine the impact of general knowledge about firms on the sentiment measurement of texts by LLMs. Firstly, we compare the sentiment scores of financial texts by LLMs when the company name is explicitly included in the prompt versus when it is not. We define and quantify company-specific bias as the difference between these scores. Next, we construct an economic model to theoretically evaluate the impact of sentiment bias on investor behavior. This model helps us understand how biased LLM investments, when widespread, can distort stock prices. This implies the potential impact on stock prices if investments driven by biased LLMs become dominant in the future. Finally, we conduct an empirical analysis using Japanese financial text data to examine the relationship between firm-specific sentiment bias, corporate characteristics, and stock performance.

Keywords: LLM Bias, Financial Sentiment Analysis, Economic Modeling, Investor Behavior, Corporate Characteristics

Complexity vs Empirical Score

• Math Complexity: 3.5/10
• Empirical Rigor: 7.5/10
• Quadrant: Street Traders
• Why: The paper employs a straightforward difference-in-means approach to quantify bias and uses a basic economic equilibrium model, indicating low-to-moderate mathematical complexity. However, it demonstrates high empirical rigor through the use of real-world Japanese financial text data, systematic testing across multiple LLMs, and an analysis linking sentiment bias to actual stock performance.

  flowchart TD
    Goal["Research Goal:<br>Evaluate LLM Sentiment Bias<br>in Financial Texts"]

    Method1["Method 1: Prompt Comparison<br>Score Texts With vs Without<br>Company Name"]
    Bias_Def["Define Bias = Score Difference"]
    Model["Method 2: Economic Model<br>Simulate Investor Behavior"]
    Data["Data: Japanese Financial<br>Texts & Stock Data"]

    Comp["Analysis: Compare Bias<br>vs Firm Characteristics &<br>Stock Performance"]

    Out1["Outcome: Quantified<br>Company-specific Bias"]
    Out2["Outcome: Theoretical Impact<br>on Stock Prices"]
    Out3["Outcome: Empirical Link<br>between Bias, Firms &<br>Performance"]

    Goal --> Method1
    Method1 --> Bias_Def
    Method1 --> Model
    Goal --> Data
    Data --> Comp
    Bias_Def --> Comp
    Model --> Out2
    Comp --> Out3
    Bias_Def --> Out1