Paper: arXiv 2507.20957

Authors: Hoyoung Lee, Junhyuk Seo, Suhwan Park, Junhyeong Lee, Wonbin Ahn, Chanyeol Choi, Alejandro Lopez-Lira, Yongjae Lee

Abstract

In finance, Large Language Models (LLMs) face frequent knowledge conflicts arising from discrepancies between their pre-trained parametric knowledge and real-time market data. These conflicts are especially problematic in real-world investment services, where a model’s inherent biases can misalign with institutional objectives, leading to unreliable recommendations. Despite this risk, the intrinsic investment biases of LLMs remain underexplored. We propose an experimental framework to investigate emergent behaviors in such conflict scenarios, offering a quantitative analysis of bias in LLM-based investment analysis. Using hypothetical scenarios with balanced and imbalanced arguments, we extract the latent biases of models and measure their persistence. Our analysis, centered on sector, size, and momentum, reveals distinct, model-specific biases. Across most models, a tendency to prefer technology stocks, large-cap stocks, and contrarian strategies is observed. These foundational biases often escalate into confirmation bias, causing models to cling to initial judgments even when faced with increasing counter-evidence. A public leaderboard benchmarking bias across a broader set of models is available at https://linqalpha.com/leaderboard

Complexity vs Empirical Score

  • Math Complexity: 4.0/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper focuses on experimental methodology and bias quantification using LLM prompting, with limited advanced mathematics, but demonstrates high empirical rigor through a structured framework, hypothetical scenario testing, and a public leaderboard benchmark.

Research Flowchart

  flowchart TD
  A["Research Goal: Quantify LLM Investment Biases in Knowledge Conflicts"] --> B["Methodology: Hypothetical Scenario Testing"]
  B --> C{"Data: Balanced vs. Imbalanced<br>Sector/Size/Momentum Arguments"}
  C --> D["Process: Extract & Measure<br>Latent Bias & Persistence"]
  D --> E["Computation: Statistical Analysis<br>of Model Responses"]
  E --> F["Key Outcomes: Distinct Model-Specific Biases"]
  F --> G["Findings: Tech, Large-Cap,<br>Contrarian Preference + Confirmation Bias"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a three-stage experimental framework to systematically elicit and quantify latent biases in LLMs for investment analysis.

Source sentence
To answer these questions, this study introduces a three-stage experimental framework designed to systematically elicit and verify LLM biases in investment analysis.

Assumes 5

  • Evidence is generated by a single LLM (Gemini-2.5-Pro) and simplified to a numerical value. data
    Source sentence
    First, our evidence was generated by a single LLM and simplified to a numerical value, an approach that may not fully capture the biases of the generator model or the complexity of real-world information.
  • All evidence is generated (no external data) to avoid generation bias. model
    Source sentence
    To neutralize this potential generation bias and ensure that observed bias are not artifacts of context sourcing, our methodology exclusively uses generated evidence for all experimental conditions.
  • Models are run at temperature τ=0.6 to balance determinism and creativity. model
    Source sentence
    All experiments were performed with the models configured at a temperature of 𝜏= 0.6, striking a balance between deterministic and creative response generation, and each experimental run was conducted three times to ensure the robustness of our results.
  • Buy and sell evidence are presented in equal proportion with identical intensity (I=5%). data
    Source sentence
    Specifically, for every stock 𝑠, buy evidence (E (𝑠) buy) and sell evi-dence (E (𝑠) sell) are generated in an equal proportion, yielding a com-prehensive dataset of |E| = 3, 416 evidence. To further isolate bias, all evidence is engineered with a uniform linguistic structure and explicitly states a fixed intensity parameter 𝐼= 5%.
Show 1 more
  • Evidence order is randomized across trials to mitigate positional bias. execution
    Source sentence
    To quantify this elicited bias, the decision task is repeated 𝑁= 10 times for each stock, with the evidence order randomized in each trial to mitigate positional bias.

Data

  • Curated set of 427 prominent stocks fundamentals public
    S&P 500 constituents continuously listed over past five years
    Source sentence
    Accordingly, our investigation is confined to a curated set of 427 prominent stocks, denoted S = {𝑠1,𝑠2, . . . ,𝑠427}. These stocks were selected for their continuous listing in the S&P 500 index over the past five years.

Costs

Not applicable The paper reports no trading or execution result.

Finds 3

  • Across most models, there is a tendency to prefer technology stocks, large‑cap stocks, and contrarian strategies. statistical
    Source sentence
    Across most models, a tendency to prefer technology stocks, large-cap stocks, and contrarian strategies is observed.
  • Size bias scores decline as company size decreases, with statistically significant gaps between largest and smallest market‑cap quantiles. statistical
    Source sentence
    Bias scores consistently decline as company size decreases. To statistically validate these observed trends, we performed independent-samples t-tests comparing the bias scores between the highest- and lowest-bias quantiles for each model (Table 2).
  • LLMs exhibit confirmation bias, clinging to initial judgments despite increasing counter‑evidence. negative
    Source sentence
    These foundational biases often escalate into confirmation bias, causing models to cling to initial judgments even when faced with increasing counter-evidence. We demonstrate how these biases lead to a stub-born refusal to revise judgments when the model is challenged with stronger counter-evidence in terms of both volume and intensity.

Checked by others

Supported by

  • Evaluating Company-specific Biases in Financial Sentiment Analysis using Large Language Models (2024)
    On the question: Do large language models show a company‑size sentiment bias in financial analysis?
    Compare the findings

    This paper: Size bias scores decline as company size decreases, with statistically significant gaps between largest and smallest market‑cap quantiles.

    That paper: Company-specific sentiment bias is present across multiple LLMs, with GPT-3.5 showing a negative evaluation of smaller firms when the company name is included.

    Both find that bias grows with size: GPT‑3.5 rates smaller firms more negatively, and bias scores drop as market‑cap decreases, with significant gaps between the largest and smallest quantiles.

Cited by 10 papers in the archive (5 by independent authors): Uncovering Representation Bias for Investment Decisions in … (extends it)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (36)

  1. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts (2023)
  2. What evidence do language models find convincing? (2024) — link
  3. Cognitive bias in decision-making with LLMs (2024) — link
  4. LLM Agents Can Be Choice-Supportive Biased Evaluators: An Empirical Study (2025)
  5. Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making (2024)
  6. What Is Seen Cannot Be Unseen: The Disruptive Effect of Knowledge Conflict on Large Language Models (2025) — link
  7. Blinded by generated contexts: How language models merge generated and retrieved contexts when knowledge conflicts? (2024) — link
  8. Exposing product bias in llm investment recommendation (2025) — link
and 28 more
  1. Evaluating company-specific biases in financial sentiment analysis using large language models — → 2411.00420
  2. Do i know this entity? knowledge awareness and hallucinations in language models (2024) — link
  3. How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Models (2025) — link
  4. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models (2024) — link
  5. Investorbench: A benchmark for financial decision-making tasks with llm-based agent — → 2412.18174
  6. LLMs analyzing the analysts: Do BERT and GPT extract more value from financial analyst reports? (2023)
  7. A multimodal foundation agent for financial trading: Tool-augmented, diversified, and generalist — → 2402.18485
  8. LLM economicus? mapping the behavioral biases of LLMs via utility theory (2024) — link
  9. Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
  10. Integrating LLM-Generated Views into Mean-Variance Optimization Using the Black-Litterman Model — → 2504.14345
  11. Decision-informed neural networks with large language model integration for portfolio optimization — → 2502.00828
  12. LLM-Powered Multi-Agent System for Automated Crypto Portfolio Management — → 2501.00826
  13. Can ChatGPT improve investment decisions? From a portfolio management perspective (2024)
  14. Large language models are inconsistent and biased evaluators (2024) — link
  15. Qwen3 technical report (2025) — link
  16. Gpt-4 technical report (2023) — link
  17. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities (2025) — link
  18. Deepseek-v3 technical report (2024) — link
  19. Can ChatGPT plan your retirement?: Generative AI and financial advice (2024)
  20. Mistral Small 3 (2025) — link
  21. Llama 4 (2025) — link
  22. TradingAgents: Multi-agents LLM financial trading framework — → 2412.20138
  23. Are llms rational investors? a study on detecting and reducing the financial bias in llms (2024) — link
  24. Behavioral Economics of AI: LLM Biases and Corrections (2025)
  25. Social Group Bias in AI Finance (2025) — link
  26. When LLMs Go Abroad: Foreign Bias in AI Financial Predictions (2025)
  27. What does chatgpt make of historical stock returns? extrapolation and miscalibration in llm stock return forecasts — → 2409.11540
  28. How ethical should AI be? how AI alignment shapes the risk preferences of llms (2024) — link

9 of 36 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.