Paper: arXiv 2312.11797
Authors: Min Dai, Yuchao Dong, Yanwei Jia, Xun Yu Zhou
Abstract
We study Merton’s expected utility maximization problem in an incomplete market, characterized by a factor process in addition to the stock price process, where all the model primitives are unknown. The agent under consideration is a price taker who has access only to the stock and factor value processes and the instantaneous volatility. We propose an auxiliary problem in which the agent can invoke policy randomization according to a specific class of Gaussian distributions, and prove that the mean of its optimal Gaussian policy solves the original Merton problem. With randomized policies, we are in the realm of continuous-time reinforcement learning (RL) recently developed in Wang et al. (2020) and Jia and Zhou (2022a, 2022b, 2023), enabling us to solve the auxiliary problem in a data-driven way without having to estimate the model primitives. Specifically, we establish a policy improvement theorem based on which we design both online and offline actor-critic RL algorithms for learning Merton’s strategies. A key insight from this study is that RL in general and policy randomization in particular are useful beyond the purpose for exploration – they can be employed as a technical tool to solve a problem that cannot be otherwise solved by mere deterministic policies. At last, we carry out both simulation and empirical studies in a stochastic volatility environment to demonstrate the decisive outperformance of the devised RL algorithms in comparison to the conventional model-based, plug-in method.
Complexity vs Empirical Score
- Math Complexity: 8.0/10
- Empirical Rigor: 6.0/10 (capped from 7.0: transaction costs not modelled; see penalty flags)
- Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 7.0)
Why this score: The paper uses advanced stochastic control (HJB, Pontryagin) and modern RL theory with convergence proofs, indicating high mathematical density. It includes simulation and empirical studies in a stochastic volatility environment, but relies on structured models without full live backtesting, placing it between theoretical and empirical rigor.
Research Flowchart
flowchart TD
A["Research Goal: Solve Data-Driven Merton Problem<br>in Incomplete Markets"] --> B{"Key Methodology: Policy Randomization"}
B --> C["Formulate Auxiliary Problem with Gaussian Policies"]
C --> D["Apply Policy Improvement Theorem<br>Derive Optimal Mean Policy"]
D --> E["Data/Inputs: Stock & Factor Prices, Volatility<br>Without Model Primitives"]
E --> F{"Computational Process: Actor-Critic RL Algorithms"}
F --> G["Online RL Learning"]
F --> H["Offline RL Learning"]
G --> I["Outcomes & Findings: <br>1. Optimal Mean Policy Solves Original Merton Problem <br>2. RL Outperforms Plug-in Methods <br>3. Policy Randomization as Technical Tool"]
H --> I
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes an auxiliary problem using Gaussian policy randomization and develops online and offline actor‑critic reinforcement learning algorithms to solve Merton's investment problem in a data‑driven way.
Source sentence
We propose an auxiliary problem in which the agent can invoke policy randomization according to a specific class of Gaussian distributions, and prove that the mean of its optimal Gaussian policy solves the original Merton problem.
Assumes 5
- The market is incomplete and includes a factor process in addition to the stock price process. market
Source sentence
characterized by a factor process in addition to the stock price process
- The agent only observes the stock price, factor value, and instantaneous volatility. data
Source sentence
has access only to the stock and factor value processes and the instantaneous volatility
- The temperature (exploration) parameter λ is strictly positive, so the learned policy is always randomized. model
Source sentence
Finally, note that our method does not permit λ “ 0 and therefore always learns a randomized policy
- A stochastic volatility model is used for the market dynamics. model
Source sentence
A stochastic volatility model is specified by µpt,xq …
Show 1 more
- Training data are generated synthetically with daily observations over 20 years. data
Source sentence
we generate a training dataset with daily data for 20 years, and each time we randomly sample a consecutive subsequence from that dataset with a length of 1 year as one episode for training
Data
- Synthetic daily market data (20‑year horizon) synthetic unknown
Source sentence
we generate a training dataset with daily data for 20 years, and each time we randomly sample a consecutive subsequence from that dataset with a length of 1 year as one episode for training
Costs
Not modeled
Finds 4
- Both RL methods outperform the plug‑in and ERM baselines on all metrics except annualized return, with RL‑NN achieving a Sharpe ratio of 0.306 versus 0.203 for the B‑H method. performance
Source sentence
We observe the two RL methods outperform the other two methods in all the metrics except the annualized return (B-H has 5.9%, slightly higher than 5.6% by RL with neural networks). In particular, RL with neural networks beats the other methods by significant margins in most criteria including the Sharpe ratio.
- RL strategies have markedly smaller maximum drawdowns and much shorter recovery times compared with the other methods. performance
Source sentence
the two RL methods have remarkably smaller maximum drawdowns during the whole period
- RL algorithms perform better than ERM when the sample size is small (fewer than 100 episodes). performance
Source sentence
Figure 1 shows that, with a small size dataset (i.e. fewer than 100 episodes), ERM performs poorly compared with our method using λ “ 0.01,0.1,1,10,100.
- RL with neural networks consistently outperforms other methods during the volatile 2000‑2010 period, including the dot‑com bust and the 2007‑2008 financial crisis. performance
Source sentence
It is clear that RL with neural networks outperforms (in terms of portfolio worth) all the others prior to around 2020
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 20 related papers). Coverage is partial, so this is neither replication nor refutation.
7 papers in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.