Paper: arXiv 2507.20957
Authors: Hoyoung Lee, Junhyuk Seo, Suhwan Park, Junhyeong Lee, Wonbin Ahn, Chanyeol Choi, Alejandro Lopez-Lira, Yongjae Lee
Abstract
In finance, Large Language Models (LLMs) face frequent knowledge conflicts arising from discrepancies between their pre-trained parametric knowledge and real-time market data. These conflicts are especially problematic in real-world investment services, where a model’s inherent biases can misalign with institutional objectives, leading to unreliable recommendations. Despite this risk, the intrinsic investment biases of LLMs remain underexplored. We propose an experimental framework to investigate emergent behaviors in such conflict scenarios, offering a quantitative analysis of bias in LLM-based investment analysis. Using hypothetical scenarios with balanced and imbalanced arguments, we extract the latent biases of models and measure their persistence. Our analysis, centered on sector, size, and momentum, reveals distinct, model-specific biases. Across most models, a tendency to prefer technology stocks, large-cap stocks, and contrarian strategies is observed. These foundational biases often escalate into confirmation bias, causing models to cling to initial judgments even when faced with increasing counter-evidence. A public leaderboard benchmarking bias across a broader set of models is available at https://linqalpha.com/leaderboard
Complexity vs Empirical Score
- Math Complexity: 4.0/10
- Empirical Rigor: 6.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper focuses on experimental methodology and bias quantification using LLM prompting, with limited advanced mathematics, but demonstrates high empirical rigor through a structured framework, hypothetical scenario testing, and a public leaderboard benchmark.
Research Flowchart
flowchart TD
A["Research Goal: Quantify LLM Investment Biases in Knowledge Conflicts"] --> B["Methodology: Hypothetical Scenario Testing"]
B --> C{"Data: Balanced vs. Imbalanced<br>Sector/Size/Momentum Arguments"}
C --> D["Process: Extract & Measure<br>Latent Bias & Persistence"]
D --> E["Computation: Statistical Analysis<br>of Model Responses"]
E --> F["Key Outcomes: Distinct Model-Specific Biases"]
F --> G["Findings: Tech, Large-Cap,<br>Contrarian Preference + Confirmation Bias"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a three-stage experimental framework to systematically elicit and quantify latent biases in LLMs for investment analysis.
Source sentence
To answer these questions, this study introduces a three-stage experimental framework designed to systematically elicit and verify LLM biases in investment analysis.
Assumes 5
- Evidence is generated by a single LLM (Gemini-2.5-Pro) and simplified to a numerical value. data
Source sentence
First, our evidence was generated by a single LLM and simplified to a numerical value, an approach that may not fully capture the biases of the generator model or the complexity of real-world information.
- All evidence is generated (no external data) to avoid generation bias. model
Source sentence
To neutralize this potential generation bias and ensure that observed bias are not artifacts of context sourcing, our methodology exclusively uses generated evidence for all experimental conditions.
- Models are run at temperature τ=0.6 to balance determinism and creativity. model
Source sentence
All experiments were performed with the models configured at a temperature of 𝜏= 0.6, striking a balance between deterministic and creative response generation, and each experimental run was conducted three times to ensure the robustness of our results.
- Buy and sell evidence are presented in equal proportion with identical intensity (I=5%). data
Source sentence
Specifically, for every stock 𝑠, buy evidence (E (𝑠) buy) and sell evi-dence (E (𝑠) sell) are generated in an equal proportion, yielding a com-prehensive dataset of |E| = 3, 416 evidence. To further isolate bias, all evidence is engineered with a uniform linguistic structure and explicitly states a fixed intensity parameter 𝐼= 5%.
Show 1 more
- Evidence order is randomized across trials to mitigate positional bias. execution
Source sentence
To quantify this elicited bias, the decision task is repeated 𝑁= 10 times for each stock, with the evidence order randomized in each trial to mitigate positional bias.
Data
- Curated set of 427 prominent stocks fundamentals public
Source sentence
Accordingly, our investigation is confined to a curated set of 427 prominent stocks, denoted S = {𝑠1,𝑠2, . . . ,𝑠427}. These stocks were selected for their continuous listing in the S&P 500 index over the past five years.
Costs
Not applicable
Finds 3
- Across most models, there is a tendency to prefer technology stocks, large‑cap stocks, and contrarian strategies. statistical
Source sentence
Across most models, a tendency to prefer technology stocks, large-cap stocks, and contrarian strategies is observed.
- Size bias scores decline as company size decreases, with statistically significant gaps between largest and smallest market‑cap quantiles. statistical
Source sentence
Bias scores consistently decline as company size decreases. To statistically validate these observed trends, we performed independent-samples t-tests comparing the bias scores between the highest- and lowest-bias quantiles for each model (Table 2).
- LLMs exhibit confirmation bias, clinging to initial judgments despite increasing counter‑evidence. negative
Source sentence
These foundational biases often escalate into confirmation bias, causing models to cling to initial judgments even when faced with increasing counter-evidence. We demonstrate how these biases lead to a stub-born refusal to revise judgments when the model is challenged with stronger counter-evidence in terms of both volume and intensity.
Checked by others
Supported by
- Evaluating Company-specific Biases in Financial Sentiment Analysis using Large Language Models
Compare the findings
This paper: Size bias scores decline as company size decreases, with statistically significant gaps between largest and smallest market‑cap quantiles.
That paper: Company-specific sentiment bias is present across multiple LLMs, with GPT-3.5 showing a negative evaluation of smaller firms when the company name is included.
10 papers in the archive : Uncovering Representation Bias for Investment Decisions in …
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (36)
- Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts (2023)
- What evidence do language models find convincing? (2024) — link
- Cognitive bias in decision-making with LLMs (2024) — link
- LLM Agents Can Be Choice-Supportive Biased Evaluators: An Empirical Study (2025)
- Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making (2024)
- What Is Seen Cannot Be Unseen: The Disruptive Effect of Knowledge Conflict on Large Language Models (2025) — link
- Blinded by generated contexts: How language models merge generated and retrieved contexts when knowledge conflicts? (2024) — link
- Exposing product bias in llm investment recommendation (2025) — link
and 28 more
- Evaluating company-specific biases in financial sentiment analysis using large language models — → 2411.00420
- Do i know this entity? knowledge awareness and hallucinations in language models (2024) — link
- How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Models (2025) — link
- Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models (2024) — link
- Investorbench: A benchmark for financial decision-making tasks with llm-based agent — → 2412.18174
- LLMs analyzing the analysts: Do BERT and GPT extract more value from financial analyst reports? (2023)
- A multimodal foundation agent for financial trading: Tool-augmented, diversified, and generalist — → 2402.18485
- LLM economicus? mapping the behavioral biases of LLMs via utility theory (2024) — link
- Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
- Integrating LLM-Generated Views into Mean-Variance Optimization Using the Black-Litterman Model — → 2504.14345
- Decision-informed neural networks with large language model integration for portfolio optimization — → 2502.00828
- LLM-Powered Multi-Agent System for Automated Crypto Portfolio Management — → 2501.00826
- Can ChatGPT improve investment decisions? From a portfolio management perspective (2024)
- Large language models are inconsistent and biased evaluators (2024) — link
- Qwen3 technical report (2025) — link
- Gpt-4 technical report (2023) — link
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities (2025) — link
- Deepseek-v3 technical report (2024) — link
- Can ChatGPT plan your retirement?: Generative AI and financial advice (2024)
- Mistral Small 3 (2025) — link
- Llama 4 (2025) — link
- TradingAgents: Multi-agents LLM financial trading framework — → 2412.20138
- Are llms rational investors? a study on detecting and reducing the financial bias in llms (2024) — link
- Behavioral Economics of AI: LLM Biases and Corrections (2025)
- Social Group Bias in AI Finance (2025) — link
- When LLMs Go Abroad: Foreign Bias in AI Financial Predictions (2025)
- What does chatgpt make of historical stock returns? extrapolation and miscalibration in llm stock return forecasts — → 2409.11540
- How ethical should AI be? how AI alignment shapes the risk preferences of llms (2024) — link
9 of 36 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.