Paper: arXiv 2511.03628
Authors: Haofei Yu, Fenghai Li, Jiaxuan You
Abstract
Large language models (LLMs) achieve strong performance across benchmarks–from knowledge quizzes and math reasoning to web-agent tasks–but these tests occur in static settings, lacking real dynamics and uncertainty. Consequently, they evaluate isolated reasoning or problem-solving rather than decision-making under uncertainty. To address this, we introduce LiveTradeBench, a live trading environment for evaluating LLM agents in realistic and evolving markets. LiveTradeBench follows three design principles: (i) Live data streaming of market prices and news, eliminating dependence on offline backtesting and preventing information leakage while capturing real-time uncertainty; (ii) a portfolio-management abstraction that extends control from single-asset actions to multi-asset allocation, integrating risk management and cross-asset reasoning; and (iii) multi-market evaluation across structurally distinct environments–U.S. stocks and Polymarket prediction markets–differing in volatility, liquidity, and information flow. At each step, an agent observes prices, news, and its portfolio, then outputs percentage allocations that balance risk and return. Using LiveTradeBench, we run 50-day live evaluations of 21 LLMs across families. Results show that (1) high LMArena scores do not imply superior trading outcomes; (2) models display distinct portfolio styles reflecting risk appetite and reasoning dynamics; and (3) some LLMs effectively leverage live signals to adapt decisions. These findings expose a gap between static evaluation and real-world competence, motivating benchmarks that test sequential decision making and consistency under live uncertainty.
Complexity vs Empirical Score
- Math Complexity: 3.5/10
- Empirical Rigor: 6.0/10 (capped from 8.0: transaction costs not modelled; see penalty flags)
- Quadrant: Street Traders — empirical and applied, lower math complexity (effective empirical rigor capped from 8.0)
Why this score: The paper is focused on building and evaluating a live trading environment with real-world data streams and portfolio management, with minimal advanced mathematical theory. Its empirical rigor is high due to live data usage, multi-market evaluation, and deployment of 21 LLMs over 50 days.
Research Flowchart
flowchart TD A["Research Goal: Evaluate LLMs in dynamic, real-world decision-making<br>under uncertainty to bridge the gap between static benchmarks<br>and live trading competence."] --> B["LiveTradeBench Design Principles:<br>1. Live Data Streaming (Prices/News)<br>2. Portfolio-Management Abstraction (Multi-Asset/Risk)<br>3. Multi-Market Evaluation (Stocks/Prediction Markets)"] B --> C["Methodology: Live Evaluation<br>- 21 LLMs (across families)<br>- 50-Day Live Trading<br>- U.S. Stocks & Polymarket"] C --> D["Computational Process:<br>1. Agent observes live market state<br>2. Outputs percentage allocations<br>3. Executes multi-asset risk-return balancing"] D --> E["Key Findings:<br>1. High benchmark scores ≠ Trading alpha<br>2. Distinct portfolio styles (risk appetite)<br>3. LLMs adapt to live signals, exposing static vs. real-world competence gap."]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes LiveTradeBench, a live multi‑market trading environment for evaluating LLM‑based agents in realistic portfolio‑management tasks.
Source sentence
we introduce LiveTradeBench, a live trading environment for evaluating LLM agents in realistic and evolving markets.
Assumes 4
- Trades have negligible market impact because assets are highly liquid. market
Source sentence
Since we focus on highly liquid assets such as Nvidia (NVDA) stocks and trending Polymarket markets to make up the portfolio, the agent’s individual trades exert negligible influence on real‑world prices. This assumption justifies modeling the simulated transition function T as closely aligned with the real world.
- Default action space is long‑only (no short positions). model
Source sentence
By default, we assume a long‑only setting where a(i)_t ≥0 for all i.
- Observation and action spaces are limited by LLM context length. data
Source sentence
The current framework constrains both the observation and action spaces due to the limited context length of existing LLMs.
- Transaction fees, bid‑ask spreads, and liquidity constraints are ignored. execution
Source sentence
Our current environment and evaluation do not account for transaction fees, bid–ask spreads, liquidity constraints, or other real‑world trading frictions.
Data
- Live market data (U.S. stocks) market-prices unknown
Source sentence
LiveTradeBench streams live market data, financial news, and social signals, eliminating the dependence on offline backtesting and thereby avoiding information leakage from the root while capturing real‑world uncertainty and feedback.
- Polymarket prediction markets text-news unknown
Source sentence
LiveTradeBench streams live market data, financial news, and social signals, eliminating the dependence on offline backtesting and thereby avoiding information leakage from the root while capturing real‑world uncertainty and feedback.
Costs
Not modeled
Source sentence
Our current environment and evaluation do not account for transaction fees, bid–ask spreads, liquidity constraints, or other real‑world trading frictions.
Finds 3
- LMArena benchmark scores have virtually no correlation with trading returns (Spearman 0.054 for stocks, –0.38 for Polymarket). performance
Source sentence
In the stock market, the Spearman correlation between LMArena scores and cumulative returns is only 0.054—virtually no relationship. In Polymarket, the correlation drops to –0.38, meaning models with higher general language ability often perform worse in the dynamic market.
- Models exhibit distinct portfolio‑management styles, e.g., GPT‑5 diversifies across many stocks while Llama4‑Scout keeps a high cash ratio. performance
Source sentence
Llama4‑Scout maintains a persistently high cash ratio (above 20%), reflecting a more cautious liquidity stance.
- Reasoning‑focused LLMs do not outperform others and show higher volatility, indicating over‑adjustment. negative
Source sentence
do not outperform others in trading performance. Instead, they exhibit substantially higher volatility (>140 in Polymarket), implying over‑adjustment during the decision process.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 24 related papers). Coverage is partial, so this is neither replication nor refutation.
4 papers in the archive : Can Blindfolded LLMs Still Trade? An Anonymization-First …
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (106)
- Scaling Learning Algorithms Towards AI (2007)
- A Fast Learning Algorithm for Deep Belief Nets (2006)
- A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist — → 2402.18485
- Chatbot arena: An open platform for evaluating llms by human preference (2024)
- FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (2024) — link
- Adaptive and Explainable Margin Trading via Large Language Models on Portfolio Management (2024) — link
- Automate Strategy Finding with LLM in Quant investment — → 2409.06289
- Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning — → 2301.10724
and 98 more
- Mastering Pair Trading with Risk-Aware Recurrent Reinforcement Learning — → 2304.00364
- Deep Reinforcement Learning for Active High Frequency Trading (2021) — link
- Agent Trading Arena: A Study on Numerical Understanding in LLM-Based Agents — → 2502.17967
- Will LLMs be Professional at Fund Investment? DeepFund: A Live Arena Perspective (2025) — link
- Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking (2025) — link
- INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent — → 2412.18174
- Deep Reinforcement Learning Agents for Strategic Production Policies in Microeconomic Market Simulations (2024) — link
- Robot See, Robot Do: Imitation Reward for Noisy Financial Environments — → 2411.08637
- Sotopia: Interactive evaluation for social intelligence in language agents (2023) — link
- Swe-bench: Can language models resolve real-world github issues? (2023) — link
- Webarena: A realistic web environment for building autonomous agents (2023) — link
- ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent (2023) — link
- Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents (2024) — link
- Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks (2025) — link
- Tree Search for Language Model Agents (2024) — link
- Webvoyager: Building an end-to-end web agent with large multimodal models (2024) — link
- Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments (2024)
- Visualwebarena: Evaluating multimodal agents on realistic visual web tasks (2024) — link
- Followbench: A multi-level fine-grained constraints following benchmark for large language models (2023) — link
- Training verifiers to solve math word problems (2021) — link
- Instruction-following evaluation for large language models (2023) — link
- Generalizing Verifiable Instruction Following (2025)
- Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings (2025) — link
- OpenCompass: A Universal Evaluation Platform for Foundation Models (2023)
- Deep learning (2016)
- Gpqa: A graduate-level google-proof q&a benchmark (2024)
- Humanity’s last exam (2025) — link
- Measuring massive multitask language understanding (2020) — link
- UQ: Assessing Language Models on Unsolved Questions (2025) — link
- RealTime QA: What’s the Answer Right Now? (2022) — link
- Kimi k2: Open agentic intelligence (2025) — link
- Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning (2025) — link
- Qwen3 technical report (2025) — link
- TradingAgents: Multi-agents LLM financial trading framework — → 2412.20138
- Trading-r1: Financial trading with llm reasoning via reinforcement learning — → 2509.11420
- Chain-of-thought prompting elicits reasoning in large language models (2022)
- Can Large Language Models Trade? Testing Financial Theories with LLM Agents in Market Simulations — → 2504.10789
- Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena (2023) — link
- When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments — → 2407.18957
- To Trade or Not to Trade: An Agentic Approach to Estimating Market Risk Improves Trading Decisions — → 2507.08584
- Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents (2025) — link
- TradingAgent: Multi-Agent Financial Decision-Making with LLMs — → 2412.20138
- AlphaAgent: LLM-Driven Alpha Mining with Regularized Exploration to Counteract Alpha Decay (2025) — link
- TradingGroup: A Multi-Agent Trading System with Self-Reflection and Data-Synthesis (2025) — link
- TradingAgents: Multi-Agents LLM Financial Trading Framework — → 2412.20138
- Can LLM-based Financial Investing Strategies Outperform the Market in Long Run? — → 2505.07078
- A deep Q-learning portfolio management framework for the cryptocurrency market (2020) — link
- Reinforcement Learning for Quantitative Trading (2021) — link
- Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States (2020) — link
- A multi-agent prediction market based on partially observable stochastic game (2011) — link
- Minute-by-Minute: Financial Markets’ Reaction to the 2020 U.S. Election (2024) — link
- Betting market efficiency and prediction in binary choice models (2022) — link
- Factor investing: alpha concentration versus diversification (2021) — link
- Combining Transformer based Deep Reinforcement Learning with Black-Litterman Model for Portfolio Optimization — → 2402.16609
- AutoAlpha: an Efficient Hierarchical Evolutionary Algorithm for Mining Alpha Factors in Quantitative Investment (2020) — link
- The Evolution of Alpha in Finance Harnessing Human Insight and LLM Agents — → 2505.14727
- FinWorld: An All-in-One Open-Source Platform for End-to-End Financial AI Research and Deployment (2025) — link
- Simulating Financial Market via Large Language Model based Agents (2024) — link
- Trading-R1: Reinforcement-Guided Large Language Models for Financial Markets — → 2509.11420
- ContestTrade: A Multi-Agent Trading System Based on Internal Contest Mechanism — → 2508.00554
- TradingGroup: Self-Reflective Multi-Agent System for Financial Markets (2025) — link
- FinBench: Benchmarking LLMs for Financial Decision-Making (2024)
- FinAgent: Evaluating Financial Reasoning in Large Language Models (2025) — link
- Benchmark Contamination and Information Leakage in Financial LLMs (2024)
- TradingGPT: Multi-Agent System with Layered Memory and Distinct Characters for Enhanced Financial Trading Performance — → 2309.03736
- React: Synergizing reasoning and acting in language models (2022)
- Claude 3.7 Sonnet and Claude Code (2025)
- The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation (2025)
- Llama 3.3 (2025)
- GPT-5 is here (2025)
- Introducing GPT-4.1 in the API (2025)
- Introducing OpenAI o3 and o4-mini (2025)
- DeepSeek-V3.1 Release (2025)
- Qwen2.5 Technical Report (2024) — link
- Introducing Claude 4 (2025)
- Claude Opus 4.1 (2025)
- Grok 3 Beta — The Age of Reasoning Agents (2025)
- Grok 4 (2025)
- Gpt-4o system card (2024) — link
- FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design — → 2311.13743
- Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models — → 2402.03659
- FinMem: Evaluating Memory and Bias in Financial LLM Benchmarks (2024) — link
- StockSim: A Dual-Mode Order-Level Simulator for Evaluating Multi-Agent LLMs in Financial Markets (2025) — link
- A New DAPO Algorithm for Stock Trading (2025) — link
- FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading — → 2502.11433
- FinLLM: A Systematic Evaluation Framework for Financial Large Language Models (2025)
- LiveCodeBench: Evaluating LLMs in Real-World Coding and Execution Environments (2024) — link
- EvalAgent: Dynamic Evaluation of Autonomous LLM Agents (2025) — link
- TiMi: Rationality-Driven Agentic System for Quantitative Financial Trading (2025) — link
- MountainLion: A Multi-Modal LLM-Based Agent System for Interpretable and Adaptive Financial Trading — → 2507.20474
- The Statistics of Sharpe Ratios (2002)
- The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality (2014)
- Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering (2022) — link
- Finsearchcomp: Towards a realistic, expert-level evaluation of financial search and reasoning (2025) — link
- StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets? (2025) — link
- Fineval: A chinese financial domain knowledge evaluation benchmark for large language models (2023) — link
- Finqa: A dataset of numerical reasoning over financial data (2021) — link
- When flue meets flang: Benchmarks and large pre-trained language model for financial domain (2022) — link
24 of 106 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.