Certifying Reward Alignment in Portfolio Reinforcement Learning

Paper: arXiv 2610.34153

Authors: Matin Tavakoli, Seyedreza Tavakoli, Faezeh Salmani, Amir Amini, Mohammad Hosein Rohban, Gholamali Aminian, Ali Habibnia Rohban

Abstract

Financial reinforcement learning often translates an investment objective into an additive risk-adjusted reward. An agent can optimise that reward accurately yet implement a different financial preference. We develop a decision audit for terminal mean-variance equilibrium that separates reward misspecification from learning error. In a finite exogenous-state market, we bound the gain from changing one action using actor, reward and continuation-critic residuals. The certificate ignores state-constant critic offsets, accommodates convex action constraints, and has an independent-sample extension that does not require the equilibrium policy. A continuation-aware actor-critic method estimates the future-policy correction from samples. Across nine controlled markets and twenty seeds, it reduces policy distance by and one-step deviation gain by relative to a matched myopic-reward network. A separate PPO comparison tests the same reward intervention within a standard optimiser. Historical tests cross seven covariance estimators and 10, 30 and 49 US industry portfolios over 432 months: joint reward calibration preserves each fitted mean-variance target, but does not improve realised investment performance. Together, these results provide a way to quantify whether a learned decision implements its stated financial objective, separately from how well that objective performs in a backtest.

Complexity vs Empirical Score

• Math Complexity: 8.0/10
• Empirical Rigor: 7.0/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a mathematically dense framework for certifying reward alignment in RL, involving complex derivations and theoretical bounds. It supports this with rigorous empirical testing across controlled markets and historical data, demonstrating a strong blend of theoretical depth and practical validation. The novelty lies in the decision audit framework and continuation-aware learning for reward alignment.

Research Flowchart

  flowchart TD
    A[Research Goal: Certify Reward Alignment in Portfolio RL] --> B{Methodology: Decision Audit for Mean-Variance Equilibrium};
    B --> C[Inputs: Finite Exogenous-State Market, Convex Action Constraints];
    C --> D{Computational Process: Actor, Reward, & Continuation-Critic Residuals};
    D --> E[Computational Process: Continuation-Aware Actor-Critic (Future-Policy Correction)];
    E --> F[Validation 1: Controlled Markets (9 markets, 20 seeds)];
    E --> G[Validation 2: Historical Tests (7 Covariance Estimators, US Industry Portfolios, 432 months)];
    F & G --> H[Key Findings: Quantify Reward Misspecification vs. Learning Error; Improved Policy Distance & One-Step Deviation Gain (Continuation-Aware AC); Joint Reward Calibration Preserves Targets, but No Improved Realized Performance];