Monte Carlo policy gradient, usually called REINFORCE after Williams (1992), is policy optimisation with nothing else attached. Parameterise a stochastic policy, run it to the end of an episode, and move the parameters along the log-probability gradient of each action taken, weighted by the return that followed. The gradient estimate is unbiased and needs no value function, no bootstrapping and no environment model. It is also high-variance: the only signal about an action’s quality is the full noisy return of its trajectory.

One corpus paper carries the exact tag “Monte Carlo Policy Gradient (MCPG)”. The ranked list widens to the policy-gradient family (13 papers); DDPG and TD3, deterministic actor-critic methods, are tagged separately but appear in the comparisons below.

Variance, and what the finance papers do about it

The textbook remedies are a baseline subtracted from the return (any state-dependent baseline keeps the gradient unbiased and can cut variance sharply), reward-to-go, and many episodes per update. Actor-critic methods go further, replacing the Monte Carlo return with a learned value estimate, trading variance for bias; PPO adds a clipped objective so no single update moves the policy too far.

The corpus has one paper whose whole contribution is this trade-off. QuantFactor REINFORCE (math 8.5, rigor 7.2) argues that in alpha-factor mining, where the “environment” is the deterministic token-by-token construction of a formula, transitions are Dirac and all variance comes from the policy, making REINFORCE with a purpose-built baseline a better fit than PPO. With information-ratio reward shaping it reports a 3.83% gain in correlation with returns over prior mining methods. The lesson is portable: the more deterministic the environment given the action, the less MCPG’s variance matters.

Why episodic finance problems suit it

MCPG needs episodes that end; much of finance is episodic. Hedging an option runs to expiry and is scored on terminal P&L. Liquidating a position runs to a deadline. Pricing a share-repurchase contract runs to an exit time. In these problems the informative reward is sparse or terminal, exactly where bootstrapped value estimates struggle and a Monte Carlo return is the honest signal.

The anchor paper is Deep Reinforcement Learning Algorithms for Option Hedging (math 7.5, rigor 6.0). It compares eight algorithms on one dynamic-hedging task: MCPG, PPO, four deep Q-learning variants and two DDPG variants, against a Black–Scholes delta-hedge baseline on GJR-GARCH(1,1) simulated paths. MCPG scored best on the root semi-quadratic penalty, PPO second, and MCPG alone beat the delta hedge within the allotted compute budget, which the authors attribute to reward sparsity. Two caveats sit in the paper’s own framing: the ranking is conditional on a compute budget, and the data are simulated; code is available, a real-market test is not.

Policy gradient learning methods for stochastic control with exit time (rigor 6.5) makes the episodic argument formally: with an exit time, randomised policies are what make the gradient computable, and the authors apply both a direct policy-gradient scheme and an actor-critic one to share-repurchase pricing, with price impact and transaction costs as extensions. On the trading side, Deep Policy Gradient Methods in Commodity Markets (rigor 7.0) and its companion Commodities Trading through Deep Policy Gradient Methods found the actor-only policy gradient beating the actor-critic variant on front-month natural gas futures, 2017–2022, with an 83% higher Sharpe than buy-and-hold. One market, one contract: suggestive, not settled.

MCPG against actor-critic, PPO, DQN and DDPG in the archive

Across the RL hub, the strongest empirical hedging results do not use MCPG. Application of Deep RL to At-the-Money S&P 500 Options Hedging (rigor 7.5) uses TD3 on intraday S&P 500 option prices, 2004–2024, walk-forward with roughly 17 years out of sample, and beats delta hedging by a margin that shrinks as the risk-awareness penalty rises. Hedging American Put Options with Deep RL (rigor 7.5) trains DDPG agents per option on calibrated stochastic-volatility paths for 80 puts across eight symbols and tests them on the realised path to maturity. Enhancing Deep Hedging of Options with Implied Volatility Surface Feedback (rigor 7.5) runs a deep policy-gradient-type algorithm with volatility-surface state on 25 years of S&P 500 option data; its edge over practitioner delta hedging grows once transaction costs are added. Deep Hedging with RL: A Practical Framework (rigor 8.5) uses a stochastic actor-critic agent in a leak-free, cost-aware environment and is unusually candid: only one policy’s test Sharpe is distinguishable from zero and its confidence interval overlaps a long-SPY benchmark.

Two policy-gradient papers extend the objective rather than the algorithm. Robust Risk-Aware Option Hedging optimises a robust risk-aware criterion for barrier options and shows the robust policies hold up when the test data-generating process differs from training. Catastrophic-risk-aware RL with EVT-based policy gradients replaces the empirical tail with an extreme-value approximation inside the gradient, applied to dynamic option hedging, with public code. On the sceptical side, Myopic Optimality (math 9.5, rigor 4.0) derives policy gradients via Malliavin calculus to argue that RL portfolio strategies book “phantom profit” relative to myopic optimisation once execution frictions are modelled; theory with no original data, but the mechanism, gradient contamination from mark-to-market rewards, is the one to watch for in any policy-gradient trading paper.

The pattern: MCPG wins clean simulated comparisons where rewards are sparse and the environment is cheap to sample; the real-data hedging results come from actor-critic and deterministic-policy-gradient methods that learn a value function from a single long history. Neither fact settles the choice; it says MCPG is the method to try first on a simulator and to be most suspicious of on a backtest.

Evaluation traps specific to policy-gradient hedging

Everything in the RL-paper checklist applies; these are the MCPG-shaped failures:

  • Reward hacking on in-sample paths. Training and testing on draws from the same GJR-GARCH or GBM generator measures how well the policy learned the generator. Ask for a test on a different generator or on realised paths; the American-put and IV-surface papers do this, the eight-algorithm comparison does not.
  • No transaction costs, or costs that do not bind. Delta hedging is only meaningfully beatable when rebalancing is costly; a policy that beats delta at zero cost has found a variance trick, not a hedge. Outperformance that grows with costs (IV-surface, American puts) is the credible shape. See transaction costs and impact.
  • Single seed. MCPG’s gradient variance makes seed dispersion larger than for value-based methods; one seed is one draw. Five seeds with spread is the minimum.
  • Compute-conditional rankings. “Best within the budget” can invert with more training, as the anchor paper acknowledges; rankings need learning curves.
  • Mean P&L instead of the distribution. Hedging is judged on the tails. Semi-quadratic penalty, CVaR or quantiles of terminal hedging error; a mean with no spread is a screenshot.
  • One regime. A hedger trained and tested in calm vol says nothing about 2008, 2020 or a vol-of-vol spike; regime dependence is fatal because the policy encodes the training regime’s volatility dynamics implicitly.

The decision

Use MCPG when the problem is genuinely episodic with sparse terminal reward, the simulator is cheap enough for thousands of episodes per update, and you will add a baseline and report seed dispersion. Switch to actor-critic or PPO when rewards are dense, episodes are long or continuing, or you are learning from one historical path. Either way, the hedging-RL papers that survive scrutiny share costs that bind, a test on paths the agent never saw, and a correctly implemented delta hedge as the comparison.

References: Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning”, Machine Learning (1992); Buehler, Gonon, Teichmann & Wood, “Deep Hedging”, Quantitative Finance (2019).

Related reading: Evaluate an RL trading paper → · Visual map of RL for trading → · Implied-volatility primer → · Regime dependence → · Transaction costs → · Hubs: Reinforcement Learning · Options & Derivatives · Volatility

Broader area: Reinforcement Learning · All topics: research topics → · Full archive: every paper →