Paper: arXiv 2610.10256

Authors: Kun Liu, Liqun Chen

Abstract

Reality may establish that an outcome occurred without identifying which evolving procedure produced it or why. This distinction matters in production ML systems whose code, configuration, and artifacts change while external feedback accumulates. We examine it in a human-directed, agent-engineered quantitative trading system, using oracle to mean an external source of realized outcomes rather than a complete correctness specification. Across one year, the account gained and outperformed a broad market index, while annual alpha was not statistically distinguishable from zero under the main retrospective specification. Retrospectively selected subperiods include adverse relative performance and conditional candidate-level weakness under declared approximate references. Engineering records document changes during the episode, and complete recommendation-to-runtime binding is unavailable. The archive does not establish a common frozen instance or a unique cause. The case motivates an outcome–diagnosis gap: outcome evidence, evaluated-object identity, and causal explanation support distinct claims. We distinguish frozen instances, pre-specified adaptive procedures, and ad-hoc development; organize archive-relative claim identifiability and an evidence hierarchy; and propose a prospective production-binding protocol. An illustrative compatible-history example shows how factual binding can resolve a recommendation’s referent without supplying its counterfactual effect. The protocol is proposed rather than prospectively validated. External feedback constrains outcome claims, while provenance and additional identification structure determine the resolution of diagnosis.

Complexity vs Empirical Score

  • Math Complexity: 4.0/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: This paper presents a novel conceptual framework for diagnosing issues in evolving ML systems within quantitative trading. While it uses a real-world case study, the mathematical formalism is moderate, focusing more on conceptual distinctions and a proposed protocol rather than complex derivations or extensive empirical validation of the protocol itself. The empirical rigor comes from the detailed analysis of a live trading account’s performance and engineering records.

Research Flowchart

  flowchart TD
    A[Research Goal/Question: Diagnosing Outcomes in Continually Evolving Agent-Engineered Systems] --> B{Key Methodology: Retrospective Analysis of Trading System}
    B --> C(Data/Inputs: 1 Year of Trading Account Data, Engineering Records)
    C --> D[Computational Processes: Performance Metrics (Alpha), Subperiod Analysis]
    D --> E{Key Findings/Outcomes: Outcome-Diagnosis Gap, Archive-Relative Claim Identifiability}
    E --> F[Proposed Protocol: Prospective Production-Binding Protocol for Causal Explanation]