Paper: arXiv 2505.07078
Authors: Weixian Waylon Li, Hyeonjun Kim, Mihai Cucuringu, Tiejun Ma
Abstract
Large Language Models (LLMs) have recently been leveraged for asset pricing tasks and stock trading applications, enabling AI agents to generate investment decisions from unstructured financial data. However, most evaluations of LLM timing-based investing strategies are conducted on narrow timeframes and limited stock universes, overstating effectiveness due to survivorship and data-snooping biases. We critically assess their generalizability and robustness by proposing FINSABER, a backtesting framework evaluating timing-based strategies across longer periods and a larger universe of symbols. Systematic backtests over two decades and 100+ symbols reveal that previously reported LLM advantages deteriorate significantly under broader cross-section and over a longer-term evaluation. Our market regime analysis further demonstrates that LLM strategies are overly conservative in bull markets, underperforming passive benchmarks, and overly aggressive in bear markets, incurring heavy losses. These findings highlight the need to develop LLM strategies that are able to prioritise trend detection and regime-aware risk controls over mere scaling of framework complexity.
Complexity vs Empirical Score
- Math Complexity: 4.5/10
- Empirical Rigor: 8.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper presents a rigorous empirical evaluation framework (FINSABER) with extensive backtesting over two decades and 100+ symbols, including bias mitigation and regime analysis, but uses relatively standard statistical and financial metrics without advanced mathematical derivations.
Research Flowchart
flowchart TD
A["Research Goal:<br>Can LLM-based Investing<br>Strategies Outperform<br>Market in Long Run?"] --> B["Data Input:<br>Long-term Historical<br>Market Data<br>20 Years / 100+ Symbols"]
B --> C["Methodology:<br>FINSABER Backtesting<br>Framework<br>Systematic Evaluation"]
C --> D["Computational Process:<br>Regime Analysis &<br>Strategies across<br>Bull/Bear Markets"]
D --> E{"Key Findings & Outcomes"}
E --> F["LLM Strategies Deteriorate<br>under broader/longer evaluation"]
E --> G["Conservative in Bull Markets<br>Underperforms Passive Benchmark"]
E --> H["Aggressive in Bear Markets<br>Incur Heavy Losses"]
H --> I["Conclusion:<br>Need for Trend Detection &<br>Regime-Aware Risk Controls"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes FINSABER, a comprehensive backtesting framework for evaluating LLM-based investing strategies over long periods and a broad symbol universe.
Source sentence
We propose FINSABER, the first comprehensive evaluation framework for LLM-based investing strategies that supports 20 years of multi-source data, including unstructured inputs such as news and filings, expands symbol coverage via unbiased selection, and mitigates survivorship, look-ahead, and data-snooping biases.
Assumes 2
- A historical average risk‑free rate of 0.03 is used for performance calculations. market
Source sentence
Thus, we set a historical average risk-free rate of 0.03 and use Moomoo’s3 standard US commission fee ($0.0049/share, minimum $0.99/order), comparable to HSBC and TradeUp4.
- Transaction costs are modeled using a flat commission of $0.0049 per share with a $0.99 minimum per order. execution
Source sentence
Thus, we set a historical average risk-free rate of 0.03 and use Moomoo’s3 standard US commission fee ($0.0049/share, minimum $0.99/order), comparable to HSBC and TradeUp4.
Data
- 20‑year multi‑source data (prices, news, filings) text-news unknown
Source sentence
We propose FINSABER, the first comprehensive evaluation framework for LLM-based investing strategies that supports 20 years of multi-source data, including unstructured inputs such as news and filings, expands symbol coverage via unbiased selection...
Costs
Partial
Flat commission $0.0049 per share with $0.99 minimum per order; used for analysis but not incorporated into headline performance numbers.
Source sentence
Thus, we set a historical average risk-free rate of 0.03 and use Moomoo’s3 standard US commission fee ($0.0049/share, minimum $0.99/order), comparable to HSBC and TradeUp4.
Finds 5
- LLM timing‑based strategies lose their reported advantage when evaluated over longer horizons and a broader cross‑section of assets. performance
Source sentence
Systematic backtests over two decades and 100+ symbols reveal that previously reported LLM advantages deteriorate significantly under broader cross-section and over a longer-term evaluation.
- LLM strategies are overly conservative in bull markets and overly aggressive in bear markets, leading to underperformance and heavy losses. performance
Source sentence
Our market regime analysis further demonstrates that LLM strategies are overly conservative in bull markets, underperforming passive benchmarks, and overly aggressive in bear markets, incurring heavy losses.
- FinMem only consistently outperforms the benchmark for TSLA; traditional rule‑based methods remain competitive or superior for other symbols. performance
Source sentence
Specifically, FinMem only consistently outperforms for TSLA, while traditional benchmarks remain competitive or superior for other symbols.
- In unbiased selection setups, traditional methods achieve higher Sharpe ratios than LLM methods (e.g., Buy and Hold Sharpe 0.703 vs. FinAgent Sharpe 0.241). performance
Source sentence
In the Volatility-based selection, traditional methods dominate even more clearly: Buy and Hold achieves the highest Sharpe (0.703)
- Statistical tests show LLM agents do not generate significant alpha; p‑values exceed 0.34, supporting the claim that their superiority does not hold under rigorous evaluation. statistical
Source sentence
neither LLM agent generates statistically significant alpha, with all measured p‑values exceeding 0.34.
Checked by others
Supported by
- AI in Investment Analysis: LLMs for Equity Stock Ratings
Compare the findings
This paper: LLM timing‑based strategies lose their reported advantage when evaluated over longer horizons and a broader cross‑section of assets.
That paper: LLMs perform better for short‑term horizons, while traditional analysts perform better over longer horizons.
9 papers in the archive : FINCOSMOS: An Open-World Evaluation Paradigm for Agents in … ; Alpha-R1: Alpha Screening with LLM Reasoning via …
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (88)
- Pairs Trading: Performance of a Relative-Value Arbitrage Rule (2006)
- Digesting Anomalies: An Investment Approach (2014)
- Mispricing Factors (2016)
- Short- and Long-Horizon Behavioral Factors (2019)
- XGBoost: A Scalable Tree Boosting System (2016)
- Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models — → 2402.03659
- FNSPID: A Comprehensive Financial News Dataset in Time Series — → 2402.06698
- Time Series Analysis, Forecasting and Control (1990)
and 80 more
- The Adaptive Markets Hypothesis: Market Efficiency from an Evolutionary Perspective (2004)
- Backtesting (2013)
- Qwen Technical Report (2023)
- The Probability of Backtest Overfitting (2015)
- Competition for attention in the ETF space (2023)
- Investments (2009)
- Bollinger on Bollinger Bands (2002)
- Survivorship bias in performance studies (1992)
- Capital Asset Prices: A Theory of Market Equilibrium Under Conditions of Risk (1964)
- Quantitative Trading: How to Build Your Own Algorithmic Trading Business (2021)
- Empirical properties of asset returns: stylized facts and statistical issues (2001)
- Momentum crashes (2016)
- Data-Snooping, Technical Trading Rule Performance, and the Bootstrap (1999)
- Defining and Dating Bull and Bear Markets: Two Centuries of Evidence (2006)
- Optimal Versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy? (2007)
- Large Language Model Agent in Financial Trading: A Survey — → 2408.06361
- TradExpert: Revolutionizing Trading with Mixture of Expert LLMs — → 2411.00782
- Does simple pairs trading still work? (2010)
- Way of the Turtle: The Secret Methods that Turned Ordinary People into Legendary Traders (2007)
- The Llama 3 Herd of Models (2024)
- Survivor bias and mutual fund performance (1996)
- Efficient capital markets (1970)
- A five-factor asset pricing model (2015)
- Can Large Language Models Beat Wall Street? Unveiling the Potential of AI in Stock Selection — → 2401.03737
- MarketSenseAI 2.0: Enhancing Stock Analysis through LLM Agents — → 2502.00415
- Learn to Rank Risky Investors: A Case Study of Predicting Retail Traders’ Behaviour and Profitability (2025) — link
- Temporal Relational Ranking for Stock Prediction (2019)
- A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist — → 2402.18485
- FinBen: A Holistic Financial Benchmark for Large Language Models (2024)
- FinRL: deep reinforcement learning framework to automate trading in quantitative finance (2022)
- Dynamic Datasets and Market Environments for Financial Reinforcement Learning (2024)
- Survivorship bias (1993)
- Mutual fund performance: An analysis of quarterly portfolio holdings (1989)
- Large Language Model Based Multi-agents: A Survey of Progress and Challenges (2024)
- GPT-InvestAR: Enhancing Stock Investment Strategies through Annual Report Analysis with Large Language Models — → 2309.03079
- Bimodal characteristic returns and predictability enhancement via machine learning (2022)
- (Re-)Imag(in)ing Price Trends (2023)
- FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting (2025)
- Adaptive Long-Short Pattern Transformer for Stock Investment Selection (2022)
- The three types of backtests (2024)
- Risk Budgeting: A Tactical Asset Allocation Approach for Retirement Reserve Funds in Morocco (2023)
- Scaling Laws for Neural Language Models (2020)
- Stock return predictability and the adaptive markets hypothesis: Evidence from century-long US data (2011)
- Enhancing the momentum strategy through deep regression (2019)
- Sentiment trading with large language models — → 2412.19245
- TradingGPT: Multi-Agent System with Layered Memory and Distinct Characters for Enhanced Financial Trading Performance — → 2309.03736
- FinRL : Deep reinforcement learning framework to automate trading in quantitative finance (2021)
- The Adaptive Markets Hypothesis: An Evolutionary Approach to Understanding Financial System Dynamics (2024)
- Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models — → 2304.07619
- Does academic research destroy stock return predictability? (2016)
- Momentum Effects in Country Equity Indices (2010)
- Noise (1986)
- The other side of value: The gross profitability premium (2013)
- On Persistence in Mutual Fund Performance (1997)
- GPT-4 Technical Report (2023)
- Optimal trading strategy during bull and bear markets for Hong Kong-listed stocks (2018)
- Quality minus junk (2019)
- Integrating Stock Features and Global Information via Large Language Models for Enhanced Stock Return Prediction — → 2310.05627
- On Persistence in Mutual Fund Performance (1997)
- Portfolio Performance Based on LLM News Scores and Related Economical Analysis (2024)
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023)
- The Sharpe Ratio (1994)
- Performance Measurement in a Downside Risk Framework (1994)
- The Profitability of Style Rotation Strategies in the United Kingdom (2005)
- LLaMA: Open and Efficient Foundation Language Models (2023)
- Equity Returns at the Turn of the Month (2008)
- The Volatility Effect: Lower Risk without Lower Return (2007)
- Where Did This ‘Bull Market’ Come From, Anyway? (2019)
- Alpha-GPT: Human-AI Interactive Alpha Mining for Quantitative Investment — → 2308.00016
- LLMFactor: Extracting Profitable Factors through Prompts for Explainable Stock Movement Prediction (2024)
- QuantAgent: Seeking Holy Grail in Trading by Self-Improving Large Language Model — → 2402.03755
- QuantBench: Benchmarking AI Methods for Quantitative Investment — → 2504.18600
- Does Trend Following Work on Stocks (2005)
- TradingAgents: Multi-Agents LLM Financial Trading Framework — → 2412.20138
- Qlib: An AI-oriented Quantitative Investment Platform (2020)
- FinGPT: Open-Source Financial Large Language Models — → 2306.06031
- FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models — → 2405.14767
- FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design — → 2311.13743
- FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (2024)
- Unveiling the Potential of Sentiment: Can Large Language Models Predict Chinese Stock Price Movements? — → 2306.14222
20 of 88 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.