ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable

Paper: arXiv 2604.25224

Authors: Sidi Chang, Peiying Zhu, Yuxiao Chen

Abstract

LLM-based financial agents increasingly produce investment rationales before the outcomes needed to evaluate them are observable. This creates a delayed-ground-truth evaluation problem: realized returns remain the eventual arbiter of investment quality, but they arrive too late and are too noisy to guide many model-development and governance decisions. LLM judges offer a tempting shortcut for pre-deployment evaluation of AI-finance systems, but unvalidated judges may reward verbosity, confidence, or rubric mimicry rather than financial judgment. This paper introduces ValueBlindBench, a preregistered agreement-gated stress-test protocol for deciding when LLM-judged investment-rationale claims are publishable, qualified, or invalid. In a controlled market-state capital-allocation prototype with 1,000 honest decision cycles and 100 preregistered adversarial controls (1,100 trajectories, 5,500 judge calls), ValueBlindBench clears the aggregate agreement gate at (\barκ_w = 0.7168) but prevents several overclaims. Lower-rank systems collapse into a tie-class, one rubric dimension fails the per-dimension gate (\texttt{constraint_awareness}, (\barκ_w = 0.2022)), single-judge rankings are family-dependent, and terse-correct rationales receive a (Δ= -2.81) rubric-point penalty relative to honest rationales. A targeted anchor-specificity probe further shows that financial constructs such as constraint awareness are operationally load-bearing. The scientific object is therefore not a leaderboard and not a claim to measure true investment skill. ValueBlindBench is a pre-calibration metrology layer for AI-finance evaluation: it governs whether a proposed LLM-judge-based investment-rationale claim is stable enough, agreed enough, and uncontaminated enough to be reported at all.

Complexity vs Empirical Score

• Math Complexity: 4.5/10
• Empirical Rigor: 8.0/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: While the paper utilizes statistical diagnostics like Krippendorff’s alpha and rigorous pre-registered testing protocols, its complexity is grounded in empirical validation of LLM-judge metrology rather than dense mathematical derivation. The high rigor score reflects the extensive execution of 5,500 judge calls across 1,100 trajectories with well-defined adversarial controls.

Research Flowchart

  flowchart TD
    A[<b>Goal:</b> Address delayed-ground-truth <br>evaluation of AI-finance agents] --> B(<b>ValueBlindBench:</b> <br>Agreement-Gated Stress-Test)
    B --> C[<b>Inputs:</b> 1,100 trajectories; <br>5,500 LLM-judge calls]
    C --> D{<b>Process:</b> Apply Protocol <br>and Measure Agreement}
    D --> E[<b>Findings:</b> Aggregate gate cleared <br>at κ=0.7168; identifying overclaims]
    E --> F[<b>Outcome:</b> Metrology layer for <br>stable, validated AI-finance reporting]

Cited References (13)

1. Advances in Financial Machine Learning (2018) — link
2. Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance (2014) — link
3. The Statistics of S harpe Ratios (2002) — link
4. Judging LLM -as-a-Judge with MT -Bench and Chatbot Arena (2023) — link
5. G -Eval: NLG Evaluation Using GPT -4 with Better Human Alignment (2023) — link
6. Holistic Evaluation of Language Models (2023) — link
7. Human Feedback is not Gold Standard (2024) — link
8. Content Analysis: An Introduction to Its Methodology (2018) — link

0 of 13 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.