Paper: arXiv 2511.03628

Authors: Haofei Yu, Fenghai Li, Jiaxuan You

Abstract

Large language models (LLMs) achieve strong performance across benchmarks–from knowledge quizzes and math reasoning to web-agent tasks–but these tests occur in static settings, lacking real dynamics and uncertainty. Consequently, they evaluate isolated reasoning or problem-solving rather than decision-making under uncertainty. To address this, we introduce LiveTradeBench, a live trading environment for evaluating LLM agents in realistic and evolving markets. LiveTradeBench follows three design principles: (i) Live data streaming of market prices and news, eliminating dependence on offline backtesting and preventing information leakage while capturing real-time uncertainty; (ii) a portfolio-management abstraction that extends control from single-asset actions to multi-asset allocation, integrating risk management and cross-asset reasoning; and (iii) multi-market evaluation across structurally distinct environments–U.S. stocks and Polymarket prediction markets–differing in volatility, liquidity, and information flow. At each step, an agent observes prices, news, and its portfolio, then outputs percentage allocations that balance risk and return. Using LiveTradeBench, we run 50-day live evaluations of 21 LLMs across families. Results show that (1) high LMArena scores do not imply superior trading outcomes; (2) models display distinct portfolio styles reflecting risk appetite and reasoning dynamics; and (3) some LLMs effectively leverage live signals to adapt decisions. These findings expose a gap between static evaluation and real-world competence, motivating benchmarks that test sequential decision making and consistency under live uncertainty.

Complexity vs Empirical Score

  • Math Complexity: 3.5/10
  • Empirical Rigor: 6.0/10 (capped from 8.0: transaction costs not modelled; see penalty flags)
  • Quadrant: Street Traders — empirical and applied, lower math complexity (effective empirical rigor capped from 8.0)

Why this score: The paper is focused on building and evaluating a live trading environment with real-world data streams and portfolio management, with minimal advanced mathematical theory. Its empirical rigor is high due to live data usage, multi-market evaluation, and deployment of 21 LLMs over 50 days.

Research Flowchart

  flowchart TD
  A["Research Goal: Evaluate LLMs in dynamic, real-world decision-making<br>under uncertainty to bridge the gap between static benchmarks<br>and live trading competence."] --> B["LiveTradeBench Design Principles:<br>1. Live Data Streaming (Prices/News)<br>2. Portfolio-Management Abstraction (Multi-Asset/Risk)<br>3. Multi-Market Evaluation (Stocks/Prediction Markets)"]
  
  B --> C["Methodology: Live Evaluation<br>- 21 LLMs (across families)<br>- 50-Day Live Trading<br>- U.S. Stocks & Polymarket"]
  
  C --> D["Computational Process:<br>1. Agent observes live market state<br>2. Outputs percentage allocations<br>3. Executes multi-asset risk-return balancing"]
  
  D --> E["Key Findings:<br>1. High benchmark scores ≠ Trading alpha<br>2. Distinct portfolio styles (risk appetite)<br>3. LLMs adapt to live signals, exposing static vs. real-world competence gap."]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes LiveTradeBench, a live multi‑market trading environment for evaluating LLM‑based agents in realistic portfolio‑management tasks.

Source sentence
we introduce LiveTradeBench, a live trading environment for evaluating LLM agents in realistic and evolving markets.

Assumes 4

  • Trades have negligible market impact because assets are highly liquid. market
    Source sentence
    Since we focus on highly liquid assets such as Nvidia (NVDA) stocks and trending Polymarket markets to make up the portfolio, the agent’s individual trades exert negligible influence on real‑world prices. This assumption justifies modeling the simulated transition function T as closely aligned with the real world.
  • Default action space is long‑only (no short positions). model
    Source sentence
    By default, we assume a long‑only setting where a(i)_t ≥0 for all i.
  • Observation and action spaces are limited by LLM context length. data
    Source sentence
    The current framework constrains both the observation and action spaces due to the limited context length of existing LLMs.
  • Transaction fees, bid‑ask spreads, and liquidity constraints are ignored. execution
    Source sentence
    Our current environment and evaluation do not account for transaction fees, bid–ask spreads, liquidity constraints, or other real‑world trading frictions.

Data

  • Live market data (U.S. stocks) market-prices unknown
    highly liquid assets such as Nvidia (NVDA) stocks
    Source sentence
    LiveTradeBench streams live market data, financial news, and social signals, eliminating the dependence on offline backtesting and thereby avoiding information leakage from the root while capturing real‑world uncertainty and feedback.
  • Polymarket prediction markets text-news unknown
    trending Polymarket markets
    Source sentence
    LiveTradeBench streams live market data, financial news, and social signals, eliminating the dependence on offline backtesting and thereby avoiding information leakage from the root while capturing real‑world uncertainty and feedback.

Costs

Not modeled A trading or execution result is reported without costs.

Source sentence
Our current environment and evaluation do not account for transaction fees, bid–ask spreads, liquidity constraints, or other real‑world trading frictions.

Finds 3

  • LMArena benchmark scores have virtually no correlation with trading returns (Spearman 0.054 for stocks, –0.38 for Polymarket). performance
    Source sentence
    In the stock market, the Spearman correlation between LMArena scores and cumulative returns is only 0.054—virtually no relationship. In Polymarket, the correlation drops to –0.38, meaning models with higher general language ability often perform worse in the dynamic market.
  • Models exhibit distinct portfolio‑management styles, e.g., GPT‑5 diversifies across many stocks while Llama4‑Scout keeps a high cash ratio. performance
    Source sentence
    Llama4‑Scout maintains a persistently high cash ratio (above 20%), reflecting a more cautious liquidity stance.
  • Reasoning‑focused LLMs do not outperform others and show higher volatility, indicating over‑adjustment. negative
    Source sentence
    do not outperform others in trading performance. Instead, they exhibit substantially higher volatility (>140 in Polymarket), implying over‑adjustment during the decision process.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 24 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 4 papers in the archive (3 by independent authors): Can Blindfolded LLMs Still Trade? An Anonymization-First … (discusses its findings)

Code

https://github.com/ulab-uiuc/live-trade-bench — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (106)

  1. Scaling Learning Algorithms Towards AI (2007)
  2. A Fast Learning Algorithm for Deep Belief Nets (2006)
  3. A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist — → 2402.18485
  4. Chatbot arena: An open platform for evaluating llms by human preference (2024)
  5. FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (2024) — link
  6. Adaptive and Explainable Margin Trading via Large Language Models on Portfolio Management (2024) — link
  7. Automate Strategy Finding with LLM in Quant investment — → 2409.06289
  8. Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning — → 2301.10724
and 98 more
  1. Mastering Pair Trading with Risk-Aware Recurrent Reinforcement Learning — → 2304.00364
  2. Deep Reinforcement Learning for Active High Frequency Trading (2021) — link
  3. Agent Trading Arena: A Study on Numerical Understanding in LLM-Based Agents — → 2502.17967
  4. Will LLMs be Professional at Fund Investment? DeepFund: A Live Arena Perspective (2025) — link
  5. Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking (2025) — link
  6. INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent — → 2412.18174
  7. Deep Reinforcement Learning Agents for Strategic Production Policies in Microeconomic Market Simulations (2024) — link
  8. Robot See, Robot Do: Imitation Reward for Noisy Financial Environments — → 2411.08637
  9. Sotopia: Interactive evaluation for social intelligence in language agents (2023) — link
  10. Swe-bench: Can language models resolve real-world github issues? (2023) — link
  11. Webarena: A realistic web environment for building autonomous agents (2023) — link
  12. ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent (2023) — link
  13. Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents (2024) — link
  14. Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks (2025) — link
  15. Tree Search for Language Model Agents (2024) — link
  16. Webvoyager: Building an end-to-end web agent with large multimodal models (2024) — link
  17. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments (2024)
  18. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks (2024) — link
  19. Followbench: A multi-level fine-grained constraints following benchmark for large language models (2023) — link
  20. Training verifiers to solve math word problems (2021) — link
  21. Instruction-following evaluation for large language models (2023) — link
  22. Generalizing Verifiable Instruction Following (2025)
  23. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings (2025) — link
  24. OpenCompass: A Universal Evaluation Platform for Foundation Models (2023)
  25. Deep learning (2016)
  26. Gpqa: A graduate-level google-proof q&a benchmark (2024)
  27. Humanity’s last exam (2025) — link
  28. Measuring massive multitask language understanding (2020) — link
  29. UQ: Assessing Language Models on Unsolved Questions (2025) — link
  30. RealTime QA: What’s the Answer Right Now? (2022) — link
  31. Kimi k2: Open agentic intelligence (2025) — link
  32. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning (2025) — link
  33. Qwen3 technical report (2025) — link
  34. TradingAgents: Multi-agents LLM financial trading framework — → 2412.20138
  35. Trading-r1: Financial trading with llm reasoning via reinforcement learning — → 2509.11420
  36. Chain-of-thought prompting elicits reasoning in large language models (2022)
  37. Can Large Language Models Trade? Testing Financial Theories with LLM Agents in Market Simulations — → 2504.10789
  38. Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena (2023) — link
  39. When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments — → 2407.18957
  40. To Trade or Not to Trade: An Agentic Approach to Estimating Market Risk Improves Trading Decisions — → 2507.08584
  41. Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents (2025) — link
  42. TradingAgent: Multi-Agent Financial Decision-Making with LLMs — → 2412.20138
  43. AlphaAgent: LLM-Driven Alpha Mining with Regularized Exploration to Counteract Alpha Decay (2025) — link
  44. TradingGroup: A Multi-Agent Trading System with Self-Reflection and Data-Synthesis (2025) — link
  45. TradingAgents: Multi-Agents LLM Financial Trading Framework — → 2412.20138
  46. Can LLM-based Financial Investing Strategies Outperform the Market in Long Run? — → 2505.07078
  47. A deep Q-learning portfolio management framework for the cryptocurrency market (2020) — link
  48. Reinforcement Learning for Quantitative Trading (2021) — link
  49. Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States (2020) — link
  50. A multi-agent prediction market based on partially observable stochastic game (2011) — link
  51. Minute-by-Minute: Financial Markets’ Reaction to the 2020 U.S. Election (2024) — link
  52. Betting market efficiency and prediction in binary choice models (2022) — link
  53. Factor investing: alpha concentration versus diversification (2021) — link
  54. Combining Transformer based Deep Reinforcement Learning with Black-Litterman Model for Portfolio Optimization — → 2402.16609
  55. AutoAlpha: an Efficient Hierarchical Evolutionary Algorithm for Mining Alpha Factors in Quantitative Investment (2020) — link
  56. The Evolution of Alpha in Finance Harnessing Human Insight and LLM Agents — → 2505.14727
  57. FinWorld: An All-in-One Open-Source Platform for End-to-End Financial AI Research and Deployment (2025) — link
  58. Simulating Financial Market via Large Language Model based Agents (2024) — link
  59. Trading-R1: Reinforcement-Guided Large Language Models for Financial Markets — → 2509.11420
  60. ContestTrade: A Multi-Agent Trading System Based on Internal Contest Mechanism — → 2508.00554
  61. TradingGroup: Self-Reflective Multi-Agent System for Financial Markets (2025) — link
  62. FinBench: Benchmarking LLMs for Financial Decision-Making (2024)
  63. FinAgent: Evaluating Financial Reasoning in Large Language Models (2025) — link
  64. Benchmark Contamination and Information Leakage in Financial LLMs (2024)
  65. TradingGPT: Multi-Agent System with Layered Memory and Distinct Characters for Enhanced Financial Trading Performance — → 2309.03736
  66. React: Synergizing reasoning and acting in language models (2022)
  67. Claude 3.7 Sonnet and Claude Code (2025)
  68. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation (2025)
  69. Llama 3.3 (2025)
  70. GPT-5 is here (2025)
  71. Introducing GPT-4.1 in the API (2025)
  72. Introducing OpenAI o3 and o4-mini (2025)
  73. DeepSeek-V3.1 Release (2025)
  74. Qwen2.5 Technical Report (2024) — link
  75. Introducing Claude 4 (2025)
  76. Claude Opus 4.1 (2025)
  77. Grok 3 Beta — The Age of Reasoning Agents (2025)
  78. Grok 4 (2025)
  79. Gpt-4o system card (2024) — link
  80. FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design — → 2311.13743
  81. Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models — → 2402.03659
  82. FinMem: Evaluating Memory and Bias in Financial LLM Benchmarks (2024) — link
  83. StockSim: A Dual-Mode Order-Level Simulator for Evaluating Multi-Agent LLMs in Financial Markets (2025) — link
  84. A New DAPO Algorithm for Stock Trading (2025) — link
  85. FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading — → 2502.11433
  86. FinLLM: A Systematic Evaluation Framework for Financial Large Language Models (2025)
  87. LiveCodeBench: Evaluating LLMs in Real-World Coding and Execution Environments (2024) — link
  88. EvalAgent: Dynamic Evaluation of Autonomous LLM Agents (2025) — link
  89. TiMi: Rationality-Driven Agentic System for Quantitative Financial Trading (2025) — link
  90. MountainLion: A Multi-Modal LLM-Based Agent System for Interpretable and Adaptive Financial Trading — → 2507.20474
  91. The Statistics of Sharpe Ratios (2002)
  92. The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality (2014)
  93. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering (2022) — link
  94. Finsearchcomp: Towards a realistic, expert-level evaluation of financial search and reasoning (2025) — link
  95. StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets? (2025) — link
  96. Fineval: A chinese financial domain knowledge evaluation benchmark for large language models (2023) — link
  97. Finqa: A dataset of numerical reasoning over financial data (2021) — link
  98. When flue meets flang: Benchmarks and large pre-trained language model for financial domain (2022) — link

24 of 106 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.