Paper: arXiv 2601.03948
Authors: Rui Sun, Yifan Sun, Sheng Xu, Li Zhao, Jing Li, Daxin Jiang, Cheng Hua, Zuo Bai
Abstract
Reinforcement Learning (RL) has enabled Large Language Models (LLMs) to achieve remarkable reasoning in domains like mathematics and coding, where verifiable rewards provide clear signals. However, extending this paradigm to financial decision is challenged by the market’s stochastic nature: rewards are verifiable but inherently noisy, causing standard RL to degenerate into reward hacking. To address this, we propose Trade-R1, a model training framework that bridges verifiable rewards to stochastic environments via process-level reasoning verification. Our key innovation is a verification method that transforms the problem of evaluating reasoning over lengthy financial documents into a structured Retrieval-Augmented Generation (RAG) task. We construct a triangular consistency metric, assessing pairwise alignment between retrieved evidence, reasoning chains, and decisions to serve as a validity filter for noisy market returns. We explore two reward integration strategies: Fixed-effect Semantic Reward (FSR) for stable alignment signals, and Dynamic-effect Semantic Reward (DSR) for coupled magnitude optimization. Experiments on different country asset selection demonstrate that our paradigm reduces reward hacking, with DSR achieving superior cross-market generalization while maintaining the highest reasoning consistency.
Complexity vs Empirical Score
- Math Complexity: 6.0/10
- Empirical Rigor: 6.0/10 (capped from 8.0: transaction costs not modelled; reported Sharpe ratio of 3.036 is implausibly high for daily or lower-frequency data; see penalty flags)
- Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 8.0)
Why this score: The paper employs advanced concepts from reinforcement learning (RL), large language models (LLMs), and structured verification metrics, indicating substantial mathematical sophistication. It also demonstrates strong empirical rigor through multi-market experiments (A-Share and US), backtest-style portfolio construction, and the proposal of specific, implementable frameworks like the Triangular Verification Protocol and semantic reward strategies.
Research Flowchart
flowchart TD
A["Research Goal:<br>Extend LLM RL to<br>Stochastic Financial Markets"] --> B["Input: Financial Docs &<br>Market Data"]
B --> C{"Process: Trade-R1 Framework<br>via Process-Level Verification"}
C --> D["Step 1: RAG-based<br>Evidence Retrieval"]
D --> E["Step 2: Triangular Consistency<br>Verification Metric"]
E --> F["Step 3: Reward Integration<br>FSR / DSR"]
F --> G["Outcome: Trade-R1 Model<br>Reduced Reward Hacking<br>DSR: Superior Generalization"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes Trade-R1, a framework that bridges verifiable rewards to stochastic financial environments via process-level reasoning verification and semantic reward strategies.
Source sentence
We presented Trade-R1, a framework bridging verifiable rewards to stochastic environments via process-level reasoning verification.
Assumes 6
- The market reward r is defined as the 10‑day forward excess return relative to the local benchmark. market
Source sentence
The market reward r is defined as the 10‑day forward excess return relative to the local benchmark (CSI 300).
- The policy model generates a reasoning chain c and a decision d, and optimization maximizes a semantic‑gated reward G(r, s). model
Source sentence
... our optimization objective maximizes the semantic‑gated reward: J(θ) = Ex∼D,y∼πθ(·|x)[G(r, s)]
- Semantic similarity score s ∈[0,1] can be computed and used as a gating factor. model
Source sentence
where G is our proposed asymmetric gating function and s ∈[0, 1] is the semantic similarity score.
- Training uses a fixed‑size Qwen3‑8B model and textual news inputs only. model
Source sentence
We employ Qwen3‑8B‑Instruct as the policy model for both markets, optimized via full‑parameter training.
Show 2 more
- The backtesting window is limited to the LLM’s knowledge cutoff to avoid data leakage. data
Source sentence
Our backtesting window is constrained by the LLM’s knowledge cutoff to prevent data leakage.
- Training is stopped at a pre‑defined step due to computational budget, not at full convergence. execution
Source sentence
Due to computational budget, we terminated training at a pre‑defined step rather than waiting for complete convergence.
Data
2025-07 – 2025-10 · A-Share (CSI-300) · equities · daily
- A‑share Market (CN) text-news unknown
Source sentence
A‑share Market (CN): Constructed from daily Chinese financial news summaries. The target universe covers all A‑share stocks.
- US Stock Market (US) text-news unknown
Source sentence
US Stock Market (US): Constructed from daily English financial news summaries. The target universe covers all stocks listed on major exchanges (e.g., NYSE, NASDAQ), explicitly excluding OTC securities to ensure liquidity.
Costs
Not modeled
Finds 5
Sharpe 3.04 · total return 37.8% · max drawdown 12.0%
- Dynamic‑effect Semantic Reward (DSR) achieves the highest semantic similarity score (0.9744) and the lowest hallucination rate (0.0012) on the A‑share test set, while also delivering strong returns. performance
Source sentence
... DSR achieves: (1) the highest Semantic Similarity Score (0.9744), indicating superior reasoning faithfulness; and (2) the lowest Hallucination Rate (0.0012), demonstrating factual grounding.
- On the out‑of‑distribution US market, DSR attains higher cumulative return (15.34%) and a better Sharpe ratio (1.951) than both Market‑Only and FSR, along with the highest similarity score (0.7768). performance
Source sentence
On the out‑of‑distribution US Market, DSR demonstrates superior cross‑market generalization: it achieves higher returns (15.34%) and a better Sharpe ratio (1.951) than both Market‑Only and FSR, alongside the highest similarity score (0.7768).
- Fixed‑effect Semantic Reward (FSR) yields a cumulative return of 39.38% on the A‑share test set, slightly lower than Market‑Only but with stable alignment signals. performance
Source sentence
... cumulative return (37.76%) is slightly lower than FSR, DSR achieves
- Unconstrained market‑only RL achieves competitive financial returns (37.62% Cum. Ret.) on A‑share but suffers from low reasoning alignment (semantic similarity 0.4369, hallucination 0.2254). performance
Source sentence
achieves competitive financial returns on the A‑Share market (37.62% Cum. Ret.) but exhibits
- The proposed Retrieval‑Augmented Verification with a Triangular Consistency metric enables accurate process supervision over long financial documents. theoretical
Source sentence
We introduced a Retrieval‑Augmented Verification protocol with a Triangular Consistency metric, assessing Factuality, Deduction, and Consistency through pairwise alignment between evidence, reasoning chains, and decisions, enabling accurate process supervision over long financial documents.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 43 related papers). Coverage is partial, so this is neither replication nor refutation.
2 papers in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (50)
- The Rise and Potential of Large Language Model Based Agents: A Survey (2023) — link
- Gpt-4 technical report (2023) — link
- Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning (2025) — link
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning (2025) — link
- Code llama: Open foundation models for code (2023) — link
- Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
- FinGPT: Open-Source Financial Large Language Models — → 2306.06031
- Efficient capital markets: A review of theory and empirical work (1970)
and 42 more
- Scaling laws for reward model overoptimization (2023)
- Scaling laws for neural language models (2020) — link
- Openai o1 system card (2024) — link
- BloombergGPT: A Large Language Model for Finance (2023) — link
- FinMem: A Performance-Enhanced LLM Trading Agent With Layered Memory and Character Design — → 2311.13743
- A multimodal foundation agent for financial trading: Tool-augmented, diversified, and generalist — → 2402.18485
- TradingAgents: Multi-agents LLM financial trading framework — → 2412.20138
- OpenAI o1 System Card (2024) — link
- DeepSeek-V3 Technical Report (2024) — link
- Trading-R1: Financial Trading with LLM Reasoning via Reinforcement Learning — → 2509.11420
- Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (2025)
- Concrete Problems in AI Safety (2016) — link
- Reward Shaping to Mitigate Reward Hacking in RLHF (2025) — link
- Reward hacking mitigation using verifiable composite rewards (2025) — link
- Constitutional AI: Harmlessness from AI Feedback (2022) — link
- Deeptrader: a deep reinforcement learning approach for risk-return balanced portfolio management with market conditions embedding (2021)
- FinRL: A deep reinforcement learning library for automated stock trading in quantitative finance (2020) — link
- Reward model ensembles help mitigate overoptimization (2023) — link
- Defining and characterizing reward gaming (2022)
- Categorizing variants of Goodhart’s Law (2018)
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (2024) — link
- Direct preference optimization: Your language model is secretly a reward model (2023)
- A Long Way to Go: The Verbosity Bias in LLM-as-a-Judge (2023) — link
- Disentangling Length from Quality in Direct Preference Optimization (2024) — link
- Understanding sycophancy in language models (2023) — link
- Discovering language model behaviors with model-written evaluations (2022) — link
- The effects of reward misspecification: Mapping and mitigating misaligned models (2022)
- Natural Emergent Misalignment from Reward Hacking in Production RL (2025) — link
- Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking (2023)
- Let’s Verify Step by Step (2023) — link
- Solving math word problems with process-and outcome-based feedback (2022) — link
- Gibbs sampling from human feedback: A provable framework for reward learning (2023)
- Stable Alignment of Large Language Models (2024) — link
- FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design — → 2311.13743
- TradingGPT: Multi-Agent System with Layered Memory and Distinct Characters for Enhanced Financial Trading Performance — → 2309.03736
- When AI Meets Finance: StockAgent, Large Language Model-based Stock Trading in Simulated Real-world Environments — → 2407.18957
- EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading — → 2309.12891
- A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist — → 2402.18485
- A lpha F in: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework (2024) — link
- FinDPO: Financial Sentiment Analysis for Algorithmic Trading through Preference Optimization of LLMs — → 2507.18417
- InvestAlign: Overcoming Data Scarcity in Aligning Large Language Models with Investor Decision-Making Processes Under Herd Behavior (2025)
- R-DDQN: Optimizing Algorithmic Trading Strategies Using a Reward Network in a Double DQN (2024)
12 of 50 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.