Multi-Dimensional Behavioral Evaluation of Agentic Stock Prediction Systems Using Large Language Model Judges with Closed-Loop Reinforcement Learning Feedback

Paper: arXiv 2605.05739

Authors: Mohammad Al Ridhawi, Mahtab Haj Ali, Hussein Al Osman

Abstract

Forecast evaluation in finance has relied on aggregate accuracy metrics and predictive-accuracy tests built on point-forecast errors. These instruments evaluate forecast outputs but cannot evaluate the process of forecast generation, which is increasingly relevant as forecasting systems become agentic, issuing forecasts through sequences of interdependent autonomous decisions whose individual quality is hidden by output-level errors. We propose a behavioral forecast-evaluation methodology that complements accuracy tests by assessing the intermediate decision process itself. Behavioral traces logged at every autonomous decision point are grouped into five-day episodes and scored along six domain-specific dimensions (regime detection, routing, adaptation, risk calibration, strategy coherence, error recovery) by an ensemble of three large language model (LLM) judges. A perturbation procedure that corrupts one dimension while leaving the other five intact confirms dimension specificity, with cross-model agreement reaching Krippendorff’s $α= 0.85$. The composite behavioral score correlates at Spearman $ρ= 0.72$ with realized 20-day Sharpe ratio from offline backtesting. Closing the loop, the framework converts deficient per-dimension scores into a credit-assigned penalty added to the Soft Actor-Critic reward. Three fine-tuning cycles, confined to the validation period, produce on the held-out 2017-2025 test period a one-day MAPE reduction from 0.61% to 0.54% (11.5% relative; $p<0.001$, Cohen’s $d=0.31$), significant under a Diebold-Mariano test of equal predictive accuracy ($\mathrm{DM}=-7.83$) and localized by a Giacomini-White conditional predictive ability test to the high-volatility regime. The methodology is application-agnostic. Results are from offline backtesting and do not address effects specific to live deployment.

Complexity vs Empirical Score

• Math Complexity: 6.5/10
• Empirical Rigor: 8.5/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper combines sophisticated Reinforcement Learning architectures and rigorous statistical validation (Diebold-Mariano, Giacomini-White, HAC corrections) with an extensive empirical backtesting pipeline across decades of S&P 500 data. It bridges theoretical agentic evaluation models with actionable, quantitative improvements to trading system performance.

Research Flowchart

  flowchart TD
    A[<b>Research Goal:</b> Evaluate Agentic Stock Prediction Process] --> B[<b>Data Input:</b> Behavioral Traces of Autonomous Decisions]
    B --> C[<b>LLM Judge Ensemble:</b> Scoring 6 Behavioral Dimensions]
    C --> D[<b>Validation:</b> Perturbation Testing & Correlation Analysis]
    D --> E[<b>Closed-Loop RL:</b> Credit Assignment to SAC Reward]
    E --> F[<b>Outcomes:</b> Improved MAPE & Enhanced Volatility Robustness]

Cited References (25)

1. Stock Market Prediction Using Node Transformer Architecture Integrated with BERT Sentiment Analysis — → 2603.05917
2. Adaptive Regime-Aware Stock Price Prediction Using Autoencoder-Gated Dual Node Transformers with Reinforcement Learning Control (2026) — link
3. Judging LLM -as-a- Judge with MT-Bench and Chatbot Arena (2023)
4. G-Eval : NLG Evaluation using GPT-4 with Better Human Alignment (2023)
5. AlpacaFarm : A Simulation Framework for Methods that Learn from Human Feedback (2024)
6. Training language models to follow instructions with human feedback (2022)
7. ReAct : Synergizing Reasoning and Acting in Language Models (2023)
8. Reflexion: Language Agents with Verbal Reinforcement Learning (2023)

1 of 25 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.