Paper: arXiv 2502.01269

Authors: Chen Ziyi, Gu Jia-wen

Abstract

We study expected utility maximization problem with constant relative risk aversion utility function in a complete market under the reinforcement learning framework. To induce exploration, we introduce the Tsallis entropy regularizer, which generalizes the commonly used Shannon entropy. Unlike the classical Merton’s problem, which is always well-posed and admits closed-form solutions, we find that the utility maximization exploratory problem is ill-posed in certain cases, due to over-exploration. With a carefully selected primary temperature function, we investigate two specific examples, for which we fully characterize their well-posedness and provide semi-closed-form solutions. It is interesting to find that one example has the well-known Gaussian distribution as the optimal strategy, while the other features the rare Wigner semicircle distribution, which is equivalent to a scaled Beta distribution. The means of the two optimal exploratory policies coincide with that of the classical counterpart. In addition, we examine the convergence of the value function and optimal exploratory strategy as the exploration vanishes. Finally, we design a reinforcement learning algorithm and conduct numerical experiments to demonstrate the advantages of reinforcement learning.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 3.5/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper provides a rigorous continuous-time mathematical framework exploring utility maximization with Tsallis entropy regularization, identifying novel ill-posedness properties and non-Gaussian strategies like the Wigner semicircle distribution. However, as a purely theoretical control paper with standard RL simulations, it lacks empirical rigor, backtesting on market data, or publicly available code.

Research Flowchart

  flowchart TD
    A[Goal: Solve CRRA Expected Utility Maximization via RL with Tsallis Entropy] --> B[Input: Complete Market Model & Primary Temperature Function]
    B --> C[Formulate Exploratory Problem with Tsallis Entropy Regularization]
    C --> D[Analyze Well-Posedness & Solve HJB Equation]
    D --> E[Derive Optimal Exploratory Policies]
    E --> F[Findings: Gaussian & Wigner Semicircle/Beta Distributions]
    F --> G[Outcomes: Convergence to Classical Merton Solution & RL Algorithm Validation]