Paper: arXiv 2505.07078

Authors: Weixian Waylon Li, Hyeonjun Kim, Mihai Cucuringu, Tiejun Ma

Abstract

Large Language Models (LLMs) have recently been leveraged for asset pricing tasks and stock trading applications, enabling AI agents to generate investment decisions from unstructured financial data. However, most evaluations of LLM timing-based investing strategies are conducted on narrow timeframes and limited stock universes, overstating effectiveness due to survivorship and data-snooping biases. We critically assess their generalizability and robustness by proposing FINSABER, a backtesting framework evaluating timing-based strategies across longer periods and a larger universe of symbols. Systematic backtests over two decades and 100+ symbols reveal that previously reported LLM advantages deteriorate significantly under broader cross-section and over a longer-term evaluation. Our market regime analysis further demonstrates that LLM strategies are overly conservative in bull markets, underperforming passive benchmarks, and overly aggressive in bear markets, incurring heavy losses. These findings highlight the need to develop LLM strategies that are able to prioritise trend detection and regime-aware risk controls over mere scaling of framework complexity.

Complexity vs Empirical Score

  • Math Complexity: 4.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper presents a rigorous empirical evaluation framework (FINSABER) with extensive backtesting over two decades and 100+ symbols, including bias mitigation and regime analysis, but uses relatively standard statistical and financial metrics without advanced mathematical derivations.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Can LLM-based Investing<br>Strategies Outperform<br>Market in Long Run?"] --> B["Data Input:<br>Long-term Historical<br>Market Data<br>20 Years / 100+ Symbols"]
  B --> C["Methodology:<br>FINSABER Backtesting<br>Framework<br>Systematic Evaluation"]
  C --> D["Computational Process:<br>Regime Analysis &<br>Strategies across<br>Bull/Bear Markets"]
  D --> E{"Key Findings & Outcomes"}
  E --> F["LLM Strategies Deteriorate<br>under broader/longer evaluation"]
  E --> G["Conservative in Bull Markets<br>Underperforms Passive Benchmark"]
  E --> H["Aggressive in Bear Markets<br>Incur Heavy Losses"]
  H --> I["Conclusion:<br>Need for Trend Detection &<br>Regime-Aware Risk Controls"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes FINSABER, a comprehensive backtesting framework for evaluating LLM-based investing strategies over long periods and a broad symbol universe.

Source sentence
We propose FINSABER, the first comprehensive evaluation framework for LLM-based investing strategies that supports 20 years of multi-source data, including unstructured inputs such as news and filings, expands symbol coverage via unbiased selection, and mitigates survivorship, look-ahead, and data-snooping biases.

Assumes 2

  • A historical average risk‑free rate of 0.03 is used for performance calculations. market
    Source sentence
    Thus, we set a historical average risk-free rate of 0.03 and use Moomoo’s3 standard US commission fee ($0.0049/share, minimum $0.99/order), comparable to HSBC and TradeUp4.
  • Transaction costs are modeled using a flat commission of $0.0049 per share with a $0.99 minimum per order. execution
    Source sentence
    Thus, we set a historical average risk-free rate of 0.03 and use Moomoo’s3 standard US commission fee ($0.0049/share, minimum $0.99/order), comparable to HSBC and TradeUp4.

Data

  • 20‑year multi‑source data (prices, news, filings) text-news unknown
    20 years · 100+ symbols
    Source sentence
    We propose FINSABER, the first comprehensive evaluation framework for LLM-based investing strategies that supports 20 years of multi-source data, including unstructured inputs such as news and filings, expands symbol coverage via unbiased selection...

Costs

Partial Costs appear only as a robustness check, a flat fee without spread or impact, or outside the headline numbers.

Flat commission $0.0049 per share with $0.99 minimum per order; used for analysis but not incorporated into headline performance numbers.

Source sentence
Thus, we set a historical average risk-free rate of 0.03 and use Moomoo’s3 standard US commission fee ($0.0049/share, minimum $0.99/order), comparable to HSBC and TradeUp4.

Finds 5

  • LLM timing‑based strategies lose their reported advantage when evaluated over longer horizons and a broader cross‑section of assets. performance
    Source sentence
    Systematic backtests over two decades and 100+ symbols reveal that previously reported LLM advantages deteriorate significantly under broader cross-section and over a longer-term evaluation.
  • LLM strategies are overly conservative in bull markets and overly aggressive in bear markets, leading to underperformance and heavy losses. performance
    Source sentence
    Our market regime analysis further demonstrates that LLM strategies are overly conservative in bull markets, underperforming passive benchmarks, and overly aggressive in bear markets, incurring heavy losses.
  • FinMem only consistently outperforms the benchmark for TSLA; traditional rule‑based methods remain competitive or superior for other symbols. performance
    Source sentence
    Specifically, FinMem only consistently outperforms for TSLA, while traditional benchmarks remain competitive or superior for other symbols.
  • In unbiased selection setups, traditional methods achieve higher Sharpe ratios than LLM methods (e.g., Buy and Hold Sharpe 0.703 vs. FinAgent Sharpe 0.241). performance
    Source sentence
    In the Volatility-based selection, traditional methods dominate even more clearly: Buy and Hold achieves the highest Sharpe (0.703)
  • Statistical tests show LLM agents do not generate significant alpha; p‑values exceed 0.34, supporting the claim that their superiority does not hold under rigorous evaluation. statistical
    Source sentence
    neither LLM agent generates statistically significant alpha, with all measured p‑values exceeding 0.34.

Checked by others

Supported by

  • AI in Investment Analysis: LLMs for Equity Stock Ratings (2024)
    On the question: Do LLM‑based investment methods retain their advantage over longer investment horizons?
    Compare the findings

    This paper: LLM timing‑based strategies lose their reported advantage when evaluated over longer horizons and a broader cross‑section of assets.

    That paper: LLMs perform better for short‑term horizons, while traditional analysts perform better over longer horizons.

    Both findings show the LLM advantage vanishes for longer horizons: A finds LLMs beat analysts only short‑term, while B reports timing‑based LLM strategies lose their edge over longer periods and a broader asset cross‑sec

Cited by 9 papers in the archive (9 by independent authors): FINCOSMOS: An Open-World Evaluation Paradigm for Agents in … (discusses its findings); Alpha-R1: Alpha Screening with LLM Reasoning via … (discusses its findings)

Code

https://github.com/waylonli/FINSABER — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (88)

  1. Pairs Trading: Performance of a Relative-Value Arbitrage Rule (2006)
  2. Digesting Anomalies: An Investment Approach (2014)
  3. Mispricing Factors (2016)
  4. Short- and Long-Horizon Behavioral Factors (2019)
  5. XGBoost: A Scalable Tree Boosting System (2016)
  6. Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models — → 2402.03659
  7. FNSPID: A Comprehensive Financial News Dataset in Time Series — → 2402.06698
  8. Time Series Analysis, Forecasting and Control (1990)
and 80 more
  1. The Adaptive Markets Hypothesis: Market Efficiency from an Evolutionary Perspective (2004)
  2. Backtesting (2013)
  3. Qwen Technical Report (2023)
  4. The Probability of Backtest Overfitting (2015)
  5. Competition for attention in the ETF space (2023)
  6. Investments (2009)
  7. Bollinger on Bollinger Bands (2002)
  8. Survivorship bias in performance studies (1992)
  9. Capital Asset Prices: A Theory of Market Equilibrium Under Conditions of Risk (1964)
  10. Quantitative Trading: How to Build Your Own Algorithmic Trading Business (2021)
  11. Empirical properties of asset returns: stylized facts and statistical issues (2001)
  12. Momentum crashes (2016)
  13. Data-Snooping, Technical Trading Rule Performance, and the Bootstrap (1999)
  14. Defining and Dating Bull and Bear Markets: Two Centuries of Evidence (2006)
  15. Optimal Versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy? (2007)
  16. Large Language Model Agent in Financial Trading: A Survey — → 2408.06361
  17. TradExpert: Revolutionizing Trading with Mixture of Expert LLMs — → 2411.00782
  18. Does simple pairs trading still work? (2010)
  19. Way of the Turtle: The Secret Methods that Turned Ordinary People into Legendary Traders (2007)
  20. The Llama 3 Herd of Models (2024)
  21. Survivor bias and mutual fund performance (1996)
  22. Efficient capital markets (1970)
  23. A five-factor asset pricing model (2015)
  24. Can Large Language Models Beat Wall Street? Unveiling the Potential of AI in Stock Selection — → 2401.03737
  25. MarketSenseAI 2.0: Enhancing Stock Analysis through LLM Agents — → 2502.00415
  26. Learn to Rank Risky Investors: A Case Study of Predicting Retail Traders’ Behaviour and Profitability (2025) — link
  27. Temporal Relational Ranking for Stock Prediction (2019)
  28. A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist — → 2402.18485
  29. FinBen: A Holistic Financial Benchmark for Large Language Models (2024)
  30. FinRL: deep reinforcement learning framework to automate trading in quantitative finance (2022)
  31. Dynamic Datasets and Market Environments for Financial Reinforcement Learning (2024)
  32. Survivorship bias (1993)
  33. Mutual fund performance: An analysis of quarterly portfolio holdings (1989)
  34. Large Language Model Based Multi-agents: A Survey of Progress and Challenges (2024)
  35. GPT-InvestAR: Enhancing Stock Investment Strategies through Annual Report Analysis with Large Language Models — → 2309.03079
  36. Bimodal characteristic returns and predictability enhancement via machine learning (2022)
  37. (Re-)Imag(in)ing Price Trends (2023)
  38. FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting (2025)
  39. Adaptive Long-Short Pattern Transformer for Stock Investment Selection (2022)
  40. The three types of backtests (2024)
  41. Risk Budgeting: A Tactical Asset Allocation Approach for Retirement Reserve Funds in Morocco (2023)
  42. Scaling Laws for Neural Language Models (2020)
  43. Stock return predictability and the adaptive markets hypothesis: Evidence from century-long US data (2011)
  44. Enhancing the momentum strategy through deep regression (2019)
  45. Sentiment trading with large language models — → 2412.19245
  46. TradingGPT: Multi-Agent System with Layered Memory and Distinct Characters for Enhanced Financial Trading Performance — → 2309.03736
  47. FinRL : Deep reinforcement learning framework to automate trading in quantitative finance (2021)
  48. The Adaptive Markets Hypothesis: An Evolutionary Approach to Understanding Financial System Dynamics (2024)
  49. Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models — → 2304.07619
  50. Does academic research destroy stock return predictability? (2016)
  51. Momentum Effects in Country Equity Indices (2010)
  52. Noise (1986)
  53. The other side of value: The gross profitability premium (2013)
  54. On Persistence in Mutual Fund Performance (1997)
  55. GPT-4 Technical Report (2023)
  56. Optimal trading strategy during bull and bear markets for Hong Kong-listed stocks (2018)
  57. Quality minus junk (2019)
  58. Integrating Stock Features and Global Information via Large Language Models for Enhanced Stock Return Prediction — → 2310.05627
  59. On Persistence in Mutual Fund Performance (1997)
  60. Portfolio Performance Based on LLM News Scores and Related Economical Analysis (2024)
  61. Toolformer: Language Models Can Teach Themselves to Use Tools (2023)
  62. The Sharpe Ratio (1994)
  63. Performance Measurement in a Downside Risk Framework (1994)
  64. The Profitability of Style Rotation Strategies in the United Kingdom (2005)
  65. LLaMA: Open and Efficient Foundation Language Models (2023)
  66. Equity Returns at the Turn of the Month (2008)
  67. The Volatility Effect: Lower Risk without Lower Return (2007)
  68. Where Did This ‘Bull Market’ Come From, Anyway? (2019)
  69. Alpha-GPT: Human-AI Interactive Alpha Mining for Quantitative Investment — → 2308.00016
  70. LLMFactor: Extracting Profitable Factors through Prompts for Explainable Stock Movement Prediction (2024)
  71. QuantAgent: Seeking Holy Grail in Trading by Self-Improving Large Language Model — → 2402.03755
  72. QuantBench: Benchmarking AI Methods for Quantitative Investment — → 2504.18600
  73. Does Trend Following Work on Stocks (2005)
  74. TradingAgents: Multi-Agents LLM Financial Trading Framework — → 2412.20138
  75. Qlib: An AI-oriented Quantitative Investment Platform (2020)
  76. FinGPT: Open-Source Financial Large Language Models — → 2306.06031
  77. FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models — → 2405.14767
  78. FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design — → 2311.13743
  79. FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (2024)
  80. Unveiling the Potential of Sentiment: Can Large Language Models Predict Chinese Stock Price Movements? — → 2306.14222

20 of 88 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.