AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models

Paper: arXiv 2602.18481

Authors: Wentao Zhang, Mingxuan Zhao, Jincheng Gao, Jieshun You, Huaiyu Jia, Yilei Zhao, Bo An, Shuo Sun

Abstract

The rapid advancement of Large Language Models (LLMs) has led to a surge of financial benchmarks, evolving from static knowledge tests to interactive trading simulations. However, current evaluations of real-time trading performance overlook a critical failure mode: severe behavioral instability in sequential decision-making under uncertainty. We empirically show that LLM-based trading agents exhibit extreme run-to-run variance, inconsistent action sequences even under deterministic decoding, and irrational action flipping across adjacent time steps. These issues stem from stateless autoregressive architectures lacking persistent action memory, as well as sensitivity to continuous-to-discrete action mappings in portfolio allocation. As a result, many existing financial trading benchmarks produce unreliable, non-reproducible, and uninformative evaluations. To address these limitations, we propose AlphaForgeBench, a principled framework that reframes LLMs as quantitative researchers rather than execution agents. Instead of emitting trading actions, LLMs generate executable alpha factors and factor-based strategies grounded in financial reasoning. This design decouples reasoning from execution, enabling fully deterministic and reproducible evaluation while aligning with real-world quantitative research workflows. Experiments across multiple state-of-the-art LLMs show that AlphaForgeBench eliminates execution-induced instability and provides a rigorous benchmark for assessing financial reasoning, strategy formulation, and alpha discovery.

Complexity vs Empirical Score

• Math Complexity: 3.0/10
• Empirical Rigor: 8.5/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper addresses a critical flaw in existing LLM-based financial trading benchmarks by identifying action instability and proposing a novel paradigm shift to evaluate LLMs as quantitative strategy coders rather than direct trading agents. It features rigorous empirical analysis across multiple models and extensive execution pipelines, though it focuses more on empirical benchmark design than complex mathematical derivations.

Research Flowchart

  flowchart TD
    A["Research Goal: Resolve Instability & Non-Reproducibility<br>in LLM Trading Agents"] --> B["Problem Identification:<br>Evaluate Action-Based Trading LLMs"]
    B --> C["Diagnosis: Action Flipping & Run-to-Run Variance<br>from Stateless Autoregressive Decoders"]
    C --> D["Proposed Framework: AlphaForgeBench"]
    E["Inputs: Financial Reasoning Prompting &<br>Market Data"] --> D
    D --> F["Methodology: Shift LLM Role from Direct Execution<br>to Generating Executable Alpha Factors/Code"]
    F --> G["Deterministic Backtesting & Evaluation<br>Decouples Reasoning from Execution"]
    G --> H["Key Outcomes: Eliminates Execution Instability,<br>Ensures Reproducibility, & Benchmarks Alpha Discovery"]

Cited References (29)

1. Finqa: A dataset of numerical reasoning over financial data (2021)
2. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance (2021) — link
3. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering (2022) — link
4. Www'18 open challenge: financial opinion mining and question answering (2018)
5. AI trading in real markets (2025)
6. Fingpt: Democratizing internet-scale data for financial large language models (2023) — link
7. Bloomberggpt: A large language model for finance (2023) — link
8. Pixiu: A comprehensive benchmark, instruction dataset and large language model for finance (2023)

3 of 29 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.