Paper: arXiv 2609.03218

Authors: Ahmed Asaad, Amr Mohamed, Yang Zhang, Omneya Abdelsalam

Abstract

Large Language Models (LLMs) increasingly use user context such as memory, profiles, and role prompts to personalize their responses. This personalization can affect evidence-based judgment: the same evidence may lead to different conclusions under different user contexts. Finance provides a high-stakes setting to study this problem because decisions often depend on interpreting long and complex documents. We test this using 3,575 SEC filings across twelve LLMs. We compare persona-conditioned retrieval, neutral retrieval, and memory-framed context to separate the effect of evidence selection from the effect of interpretation. We find that most user-context spillover comes from how models interpret the same evidence under different roles, rather than from retrieving different evidence. We then test two simple mitigation strategies: expressing the same investor mindset as a user profile instead of an assistant role, and separating evidence-based and personalized outputs. Both reduce spillover, but neither removes it completely, and their effectiveness varies substantially across models.

Complexity vs Empirical Score

  • Math Complexity: 4.0/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: This paper presents a novel and empirically rigorous study on LLM biases in financial analysis. While the mathematical complexity is moderate, the experimental design and extensive testing across multiple LLMs and SEC filings are highly commendable. The findings have significant implications for the reliable deployment of LLMs in finance.

Research Flowchart

  flowchart TD
    A[Research Goal: Impact of User Context on LLM Financial Analysis] --> B{Methodology: Compare Persona-Conditioned, Neutral, & Memory-Framed Contexts};
    B --> C[Data: 3,575 SEC Filings];
    C --> D{Computational Process: Test 12 LLMs};
    D --> E{Findings: Interpretation (not retrieval) drives context spillover};
    E --> F[Outcomes: Mitigation Strategies (user profile/separate outputs) reduce, but don't eliminate, spillover];

Cited References (48)

  1. Giving Content to Investor Sentiment: The Role of Media in the Stock Market (2007) — link
  2. Predicting Risk from Financial Reports with Regression (2009) — link
  3. When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks (2011) — link
  4. Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts (2014) — link
  5. FinBERT : Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
  6. FinQA : A Dataset of Numerical Reasoning over Financial Data (2021) — link
  7. ConvFinQA : Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering (2022) — link
  8. BloombergGPT : A Large Language Model for Finance (2023) — link
and 40 more
  1. FinGPT : Open-Source Financial Large Language Models — → 2306.06031
  2. PIXIU : A Large Language Model, Instruction Data and Evaluation Benchmark for Finance (2023) — link
  3. Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models — → 2304.07619
  4. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying about Prompt Formatting (2024) — link
  5. PromptRobust : Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (2023) — link
  6. P ro SA : Assessing and Understanding the Prompt Sensitivity of LLM s (2024) — link
  7. Persona is a Double-edged Sword: Mitigating the Negative Impact of Role-playing Prompts in Zero-shot Reasoning Tasks (2024) — link
  8. Financial Statement Analysis with Large Language Models — → 2407.17866
  9. TradingAgents : Multi-Agents LLM Financial Trading Framework (2025) — link
  10. Chat Bankman-Fried : an Exploration of LLM Alignment in Finance (2025) — link
  11. Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs (2025) — link
  12. AveniBench : Accessible and Versatile Evaluation of Finance Intelligence (2025) — link
  13. Financial Language Model Evaluation ( FLaME ) (2025) — link
  14. Are LLMs Rational Investors? A Study on the Financial Bias in LLMs (2025) — link
  15. The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models (2025) — link
  16. Position is Power: System Prompts as a Mechanism of Bias in Large Language Models ( LLMs ) (2025) — link
  17. Benchmarking Prompt Sensitivity in Large Language Models (2025) — link
  18. Beyond the Reported Cutoff: Where Large Language Models Fall Short on Financial Knowledge (2025) — link
  19. Your AI , Not Your View: The Bias of LLMs in Investment Analysis — → 2507.20957
  20. When LLMs Go Abroad: Foreign Bias in AI Financial Predictions (2025) — link
  21. Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection (2026) — link
  22. Evaluating LLMs in Finance Requires Explicit Bias Consideration — → 2602.14233
  23. Lazy Prices (2020) — link
  24. FNSPID : A Comprehensive Financial News Dataset in Time Series (2024) — link
  25. LaMP : When Large Language Models Meet Personalization (2024) — link
  26. Guided Profile Generation Improves Personalization with Large Language Models (2024) — link
  27. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) — link
  28. Stereotype or Personalization? User Identity Biases Chatbot Recommendations (2025) — link
  29. Personalization of Large Language Models: A Survey (2025) — link
  30. Portfolio Selection (1952) — link
  31. Capital Asset Prices: A Theory of Market Equilibrium under Conditions of Risk (1964) — link
  32. The Cross-Section of Expected Stock Returns (1992) — link
  33. Returns to Buying Winners and Selling Losers: Implications for Stock Market Efficiency (1993) — link
  34. Fact or Fiction: Verifying Scientific Claims (2020) — link
  35. AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web (2023) — link
  36. ContractNLI : A Dataset for Document-level Natural Language Inference for Contracts (2021) — link
  37. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (2021) — link
  38. Efficient Attentions for Long Document Summarization (2021) — link
  39. MS \textsuperscript 2 : Multi-Document Summarization of Medical Studies (2021) — link
  40. Cold Cases: An Open Caselaw Access Project Corpus (2024) — link

5 of 48 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.