Paper: arXiv 2611.06607

Authors: Henry Zhang

Abstract

Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning. In trading research the same loop is a machine for producing false discoveries: a backtest that “improves the metric” is, by default, overfit, look-ahead, a multiple-comparisons artifact, cost-free, or market drift wearing a strategy. We describe Autonomous Quant Researcher, an open-source loop built on the opposite premise: the model proposes, and only trusted code decides. Proposals are strict declarative JSON over a fixed vocabulary, so there is no model-written code to sandbox. Evaluation runs on immutable chronological stores split into discovery, validation and a final holdout that can be opened exactly once, after a durable receipt is written. Acceptance is a conjunction of absolute gates (positive edge and net P&L after costs, a positive weekly-cluster bootstrap lower bound, survival at elevated costs, per-year stability and sample floors), never a single score. Duplicates are refused before they spend a trial, and every formulation, refusal, crash and result is kept in a hash-chained, append-only ledger. We report every campaign run to date, across 0DTE options priced from traded option bars, overnight SPX verticals, daily ETF rules, technical-analysis premises, Kalshi crypto binaries, earnings options, off-exchange volume and a pre-registered alternative-data campaign. Most hypotheses fail, and we report them. Two results survive their pre-registered bars. The clearest example of the loop working is a failure: a language-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0.26 on its single holdout use. The value of such a loop lies less in what it finds than in how hard it is to make it report something that is not there.

Complexity vs Empirical Score

  • Math Complexity: 5.0/10
  • Empirical Rigor: 9.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper presents a robust and novel framework for LLM-driven quant research, focusing heavily on preventing false discoveries through rigorous empirical validation. The methodology is meticulously detailed, emphasizing reproducibility and a comprehensive reporting of all campaign outcomes, including failures.

Research Flowchart

  flowchart TD
    A[Research Goal: Automate Quantitative Finance Research while Mitigating False Discoveries] --> B{Methodology: Autonomous Quant Researcher (AQR) Loop};
    B -- Model Proposes --> C[Proposal: Strict Declarative JSON (Fixed Vocabulary)];
    C -- Trusted Code Decides --> D{Evaluation: Immutable Chronological Stores};
    D -- Stores Split Into --> D1[Discovery] & D2[Validation] & D3[Final Holdout (Open Once)];
    D1 & D2 & D3 -- Conjunction of Absolute Gates --> E{Acceptance Criteria: P&L, Bootstrap LB, Cost Survival, Stability, Sample Floors};
    E -- Ledger Records All --> F[Outcomes: Most Hypotheses Fail, Two Survive Pre-registered Bars];
    F -- Example: LLM Survivor --> G[Failure Example: Sharpe Ratio 3 -> 0.26 on Holdout];
    G --> H[Key Finding: Loop Value in Resistance to False Discoveries, Not Just Findings];