Paper: arXiv 2405.13609

Abstract

Markov decision processes (MDPs) are used to model a wide variety of applications ranging from game playing over robotics to finance. Their optimal policy typically maximizes the expected sum of rewards given at each step of the decision process. However, a large class of problems does not fit straightforwardly into this framework: Non-cumulative Markov decision processes (NCMDPs), where instead of the expected sum of rewards, the expected value of an arbitrary function of the rewards is maximized. Example functions include the maximum of the rewards or their mean divided by their standard deviation. In this work, we introduce a general mapping of NCMDPs to standard MDPs. This allows all techniques developed to find optimal policies for MDPs, such as reinforcement learning or dynamic programming, to be directly applied to the larger class of NCMDPs. Focusing on reinforcement learning, we show applications in a diverse set of tasks, including classical control, portfolio optimization in finance, and discrete optimization problems. Given our approach, we can improve both final performance and training time compared to relying on standard MDPs.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper presents a sophisticated theoretical framework mapping NCMDPs to MDPs using auxiliary state variables and reward transformations, supported by heavy mathematical notation and proofs. While it includes numerical experiments on finance portfolio optimization and control tasks, the summary focuses on the theoretical mapping and lacks specific implementation details, backtest results, or statistical metrics needed for high empirical rigor.

Research Flowchart

  flowchart TD
  Goal["Research Goal: Solve Non-Cumulative Markov Decision Processes (NCMDPs)"] --> Methodology["Methodology: Map NCMDPs to Standard MDPs"]
  Methodology --> Inputs["Inputs: Diverse Domains<br>(Control, Finance, Optimization)"]
  Inputs --> Process["Computational Process:<br>Reinforcement Learning on Mapped MDPs"]
  Process --> Outcome1["Outcome: Improved Final Performance"]
  Process --> Outcome2["Outcome: Faster Training Time"]
  Process --> Outcome3["Outcome: Unified Framework for NCMDPs"]