Paper: arXiv 2606.08285

Authors: Junyi Yao, Zihao Zheng, Baichuan Li, Jiayu Long

Abstract

Execution realism remains weakly standardized in research on large-language-model-based trading systems, limiting comparison and reproduction across backtests, simulations, and portfolio benchmarks. This article presents a PRISMA-ScR systematic scoping review with a nested execution-reproducibility evidence audit. Eight reproducible query families returned 796 query-level records; DOI/title deduplication left 687 unique records, 101 reports were sought for retrieval, and 59 full texts were recovered through direct and fallback open-access routes. Fifty-three reports are provisionally eligible for expanded evidence charting, subject to reconciliation of two blind author-validation packets. The charting framework separately evaluates point-in-time controls, temporal splits, held-out evaluation, trading costs and turnover, execution semantics, universe construction, architecture reporting, and artifact availability. Direct-trading, portfolio or alpha-construction, and benchmark studies are retained as distinct descriptive subgroups rather than pooled as a common performance estimand. Search responses, retrieval attempts, file hashes, evidence excerpts, screening decisions, and coding worksheets are archived with the review materials. Final field-level counts are intentionally withheld until author validation is complete.

Complexity vs Empirical Score

  • Math Complexity: 3.0/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: This paper offers a highly rigorous and novel audit of LLM-based trading systems, focusing on critical issues of execution realism and reproducibility. While the mathematical complexity is low, its systematic approach and clear methodology make it a valuable contribution to the field.

Research Flowchart

  flowchart TD
    A[Research Goal: Assess Execution Realism & Reproducibility in LLM-Based Trading Systems] --> B{Methodology: PRISMA-ScR Systematic Scoping Review & Evidence Audit};
    B --> C[Data/Inputs: 796 Query Records (DOI/Title Deduplicated to 687 Unique Records)];
    C --> D{Computational Processes: Screening, Retrieval (101 Sought, 59 Recovered), Charting Framework Application};
    D --> E[Charting Framework: Point-in-Time Controls, Temporal Splits, Held-out Evaluation, Trading Costs/Turnover, Execution Semantics, Universe Construction, Architecture Reporting, Artifact Availability];
    E --> F[Outcomes: 53 Provisionally Eligible Reports for Expanded Charting, Distinct Subgroups (Direct-Trading, Portfolio/Alpha-Construction, Benchmark Studies), Archived Review Materials];