Paper: arXiv 2602.09504

Authors: Sean Cao, Wei Jiang, Hui Xu

Abstract

This research explores how human-defined goals influence the behavior of Large Language Models (LLMs) through purpose-conditioned cognition. Using financial prediction tasks, we show that revealing the downstream use (e.g., predicting stock returns or earnings) of LLM outputs leads the LLM to generate biased sentiment and competition measures, even though these measures are intended to be downstream task-independent. Goal-aware prompting shifts these intermediate measures toward the disclosed downstream objective, producing in-sample overfitting. Specifically, purpose leakage improves performance on data prior to the LLM’s knowledge cutoff, but provides no advantage after the cutoff. This bias is strong enough that regularization of prompt instructions cannot fully address this form of overfitting. We further show that the bias can arise from users’ unintentional conversational context that hints at the purpose. Overall, we document that AI bias due to “seeing the goal” is not an algorithmic flaw, but stems from human accountability in research design.

Complexity vs Empirical Score

  • Math Complexity: 3.0/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: This paper presents a highly original empirical study on ‘purpose-conditioned cognition’ in LLMs, showing how disclosing downstream tasks induces single-pass in-sample overfitting (look-ahead bias) in financial sentiment extraction. The math is minimal (standard regressions and portfolio sorts), but the experimental design and empirical rigor in demonstrating the breakdown across LLM knowledge cutoffs are exceptional.

Research Flowchart

  flowchart TD
    A["Research Goal:<br>Assess impact of goal disclosure (purpose leakage) on LLM intermediate outputs"] --> B["Data & Inputs:<br>Financial text, sentiment tasks & downstream goals (stock/earnings prediction)"]
    
    B --> C["Prompting Methodology:<br>Compare Goal-Agnostic vs. Goal-Aware Prompts (including subtle user context)"]
    
    C --> D["Computational Analysis:<br>Extract intermediate measures & evaluate pre/post knowledge cutoff performance"]
    
    D --> E["Intervention Testing:<br>Apply prompt-level regularization instructions to mitigate bias"]
    
    E --> F["Key Outcomes:<br>1. Goal-aware prompts cause in-sample overfitting (no post-cutoff gain)<br>2. Prompt regularization fails to eliminate bias<br>3. AI bias stems from human research design, not algorithmic flaws"]

Cited References (39)

  1. Lookahead bias in pretrained language models (2024)
  2. The Economics of Density: Evidence From the Berlin Wall (2015) — link
  3. A new approach to predicting analyst forecast errors: Do investors overweight analyst forecasts? (2013)
  4. Risk, return, and equilibrium: Empirical tests (1973)
  5. Generative AI and asset management (2024)
  6. Total Recall? Evaluating the Macroeconomic Knowledge of Large Language Models (2025)
  7. Chronologically Consistent Large Language Models — → 2502.21206
  8. Assessing look-ahead bias in stock return predictions generated by gpt sentiment analysis — → 2309.17322
and 31 more
  1. The Memorization Problem: Can We Trust LLMs’ Economic Forecasts? — → 2504.14765
  2. What does chatgpt make of historical stock returns? extrapolation and miscalibration in llm stock return forecasts — → 2409.11540
  3. AI as Decision-Maker: Risk Preferences of LLMs (2024)
  4. Predictably unequal? The effects of machine learning on credit markets (2022)
  5. Generative AI,Managerial Expectations, and Economic Activity (2024)
  6. Generative AI and firm values (2025)
  7. ChatGPT and DeepSeek: Can they predict the stock market and macroeconomy? (2025) — link
  8. Generating inflation expectations with large language models (2025)
  9. AI and the Fed (2025)
  10. FX sentiment analysis with large language models (2025)
  11. Self-Confidence and Personal Motivation (2002)
  12. Mindful Economics: The Production, Consumption, and Value of Beliefs (2016)
  13. The Framing of Decisions and the Psychology of Choice (1981)
  14. Coarse Thinking and Persuasion (2008)
  15. Towards Understanding Sycophancy in Language Models (2023) — link
  16. AI and Perception Biases in Investments: An Experimental Study (2024)
  17. Who gets the callback? generative ai and gender bias (2025) — link
  18. Your ai, not your view: The bias of llms in investment analysis (2025)
  19. Financial Stability Implications of Generative AI: Taming the Animal Spirits — → 2510.01451
  20. Ai-powered trading, algorithmic collusion, and price efficiency (2025)
  21. When LLM go abroad: Foreign bias in ai financial predictions (2025)
  22. Social Finance in the Age of AI: A Tale of Two Platforms (2025)
  23. Coarse Thinking and Persuasion (2008)
  24. Training Language Models to Follow Instructions with Human Feedback (2022)
  25. Deep Learning (2016)
  26. Concrete Problems in AI Safety (2016) — link
  27. Behavioral Economics of AI: LLM Biases and Corrections (2026)
  28. \ldots and the Cross-Section of Expected Returns (2016)
  29. Replicating Anomalies (2020)
  30. Does Academic Research Destroy Stock Return Predictability? (2016)
  31. Predicting excess stock returns out of sample: Can anything beat the historical average? (2008)

5 of 39 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.