Paper: arXiv 2604.11477

Authors: Kun Liu, Liqun Chen

Abstract

The alignment of Multi-Agent Systems (MAS) for autonomous software engineering is constrained by evaluator epistemic uncertainty. Current paradigms, such as Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF), frequently induce model sycophancy, while execution-based environments suffer from adversarial “Test Evasion” by unconstrained agents. In this paper, we introduce an objective alignment paradigm: \textbf{Out-of-Money Reinforcement Learning (OOM-RL)}. By deploying agents into the non-stationary, high-friction reality of live financial markets, we utilize critical capital depletion as an un-hackable negative gradient. Our longitudinal 20-month empirical study (July 2024 – February 2026) chronicles the system’s evolution from a high-turnover, sycophantic baseline to a robust, liquidity-aware architecture. We demonstrate that the undeniable ontological consequences of financial loss forced the MAS to abandon overfitted hallucinations in favor of the \textbf{Strict Test-Driven Agentic Workflow (STDAW)}, which enforces a Byzantine-inspired uni-directional state lock (RO-Lock) anchored to a deterministically verified $\geq 95%$ code coverage constraint matrix. Our results show that while early iterations suffered severe execution decay, the final OOM-RL-aligned system achieved a stable equilibrium with an annualized Sharpe ratio of 2.06 in its mature phase. We conclude that substituting subjective human preference with rigorous economic penalties provides a robust methodology for aligning autonomous agents in high-stakes, real-world environments, laying the groundwork for generalized paradigms where computational billing acts as an objective physical constraint

Complexity vs Empirical Score

  • Math Complexity: 4.5/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper introduces a highly novel alignment paradigm (OOM-RL) by using actual financial capital loss as an objective gradient for multi-agent LLM systems. While the mathematical formulations are primarily conceptual and agentic rather than heavily analytical, the empirical methodology relies on an impressive 20-month live deployment demonstrating strong performance metrics (Sharpe 2.06).

Research Flowchart

  flowchart TD
    A["Research Goal: Align MAS without Human Bias or Test Evasion"] --> B["Data Input: 20-Month Live Financial Market Data (2024-2026)"]
    B --> C["OOM-RL Framework: Capital Depletion as Negative Gradient"]
    C --> D["Evolutionary Process: Baseline Sycophancy to Economic Penalty"]
    D --> E["STDAW Enforcement: Uni-directional State Lock (RO-Lock) & ≥95% Test Coverage"]
    E --> F["Key Outcome: Elimination of Hallucinations & Stable Equilibrium (Sharpe 2.06)"]