Paper: arXiv 2405.13609
Authors: Maximilian Nägele, Jan Olle, Thomas Fösel, Remmy Zen, Florian Marquardt
Abstract
Markov decision processes (MDPs) are used to model a wide variety of applications ranging from game playing over robotics to finance. Their optimal policy typically maximizes the expected sum of rewards given at each step of the decision process. However, a large class of problems does not fit straightforwardly into this framework: Non-cumulative Markov decision processes (NCMDPs), where instead of the expected sum of rewards, the expected value of an arbitrary function of the rewards is maximized. Example functions include the maximum of the rewards or their mean divided by their standard deviation. In this work, we introduce a general mapping of NCMDPs to standard MDPs. This allows all techniques developed to find optimal policies for MDPs, such as reinforcement learning or dynamic programming, to be directly applied to the larger class of NCMDPs. Focusing on reinforcement learning, we show applications in a diverse set of tasks, including classical control, portfolio optimization in finance, and discrete optimization problems. Given our approach, we can improve both final performance and training time compared to relying on standard MDPs.
Complexity vs Empirical Score
- Math Complexity: 8.5/10
- Empirical Rigor: 4.0/10
- Quadrant: Lab Rats — high math complexity, limited empirical testing
Why this score: The paper presents a sophisticated theoretical framework mapping NCMDPs to MDPs using auxiliary state variables and reward transformations, supported by heavy mathematical notation and proofs. While it includes numerical experiments on finance portfolio optimization and control tasks, the summary focuses on the theoretical mapping and lacks specific implementation details, backtest results, or statistical metrics needed for high empirical rigor.
Research Flowchart
flowchart TD Goal["Research Goal: Solve Non-Cumulative Markov Decision Processes (NCMDPs)"] --> Methodology["Methodology: Map NCMDPs to Standard MDPs"] Methodology --> Inputs["Inputs: Diverse Domains<br>(Control, Finance, Optimization)"] Inputs --> Process["Computational Process:<br>Reinforcement Learning on Mapped MDPs"] Process --> Outcome1["Outcome: Improved Final Performance"] Process --> Outcome2["Outcome: Faster Training Time"] Process --> Outcome3["Outcome: Unified Framework for NCMDPs"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a general mapping that transforms non‑cumulative Markov decision processes (NCMDPs) into standard MDPs, enabling the use of existing reinforcement‑learning methods.
Source sentence
we introduce a general mapping of NCMDPs to standard MDPs.
Assumes 5
- The analysis focuses on non‑discounted, episodic MDPs. model
Source sentence
For simplicity, we discuss only non-discounted, episodic MDPs.
- The mapping works for both deterministic and probabilistic environments. model
Source sentence
In this work, we described a mapping from a decision process with a general non‑cumulative objective (NCMDP) to a standard Markov decision process (MDP) applicable in deterministic and probabilistic settings.
- The portfolio‑optimization environment provides a history of returns and volatility measures for each index. data
Source sentence
The states contain a history of returns of each index and different volatility measures.
- Agents trade continuously on daily portfolio allocations without explicit transaction‑cost modeling. execution
Source sentence
Actions are the continuous relative portfolio allocations for each day.
Show 1 more
- The Sharpe ratio is computed as mean divided by standard deviation of simple returns. statistical
Source sentence
STD(˜r0, . . . , ˜rT−1) , where ˜rt = (Pt+1 −Pt)/Pt are the simple returns.
Data
- 11 S&P500 sector indices market-prices unknown
Source sentence
We perform experiments on an environment as described by Sood et al. (2023) where an agent trades on the 11 different S&P500 sector indices between 2006 and 2021.
Costs
Not modeled
Finds 3
- The SHARPE algorithm that directly maximizes the exact Sharpe ratio significantly outperforms both the differential Sharpe ratio and the cumulative final‑Sharpe baselines. performance
Source sentence
the SHARPE algorithm significantly outperforms the other algorithms.
- In the lunar‑lander control task, the non‑cumulative MAXVELPPO agent achieves a better trade‑off between reward and maximum speed than the cumulative FINALPPO agent. performance
Source sentence
the non‑cumulative MAXVELPPO agent can find a better trade‑off than the cumulative FINALPPO agent.
- The proposed mapping improves both final performance and training efficiency compared to standard MDP approaches. performance
Source sentence
Our approach improves both final performance and training efficiency compared to relying on standard MDPs.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 12 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (45)
- Deep Reinforcement Learning for Optimal Portfolio Allocation: A Comparative Study with Mean-Variance Optimization — → 2602.17098
- Quantum Circuit Discovery for Fault-Tolerant Logical State Preparation with Reinforcement Learning (2024) — link
- To the Max: Reinventing Reward in Reinforcement Learning (2024) — link
- Reinforcement Learning Based Quantum Circuit Optimization via ZX-Calculus (2023) — link
- Optimizing ZX-diagrams with deep reinforcement learning (2023) — link
- Cost Explosion for Efficient Reinforcement Learning Optimisation of Quantum Circuits (2023) — link
- Simultaneous discovery of quantum error correction codes and encoders with a noise-aware reinforcement learning agent (2023) — link
- Reinforcement Learning With Non-Cumulative Objective (2023) — link
and 37 more
- Faster sorting algorithms discovered using deep reinforcement learning (2023) — link
- Discovered Policy Optimisation (2022) — link
- Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimization (2022) — link
- Reachability Constrained Reinforcement Learning (2022) — link
- Application Placement in Fog Environments using Multi-Objective Reinforcement Learning with Maximum Reward Formulation (2022) — link
- Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learning (2021) — link
- Quantum circuit optimization with deep reinforcement learning (2021) — link
- Maximum Reward Formulation In Reinforcement Learning (2020) — link
- Generating Realistic Stock Market Order Streams (2020) — link
- Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning (2019) — link
- Graph-theoretic Simplification of Quantum Circuits with the ZX-calculus (2019) — link
- Optimization of Molecules via Deep Reinforcement Learning (2018) — link
- Learning dexterous in-hand manipulation (2018) — link
- Proximal Policy Optimization Algorithms (2017) — link
- Picturing Quantum Processes: A First Course in Quantum Theory and Diagrammatic Reasoning (2017) — link
- Human-level control through deep reinforcement learning (2015) — link
- Reach-avoid problems with time-varying dynamics, targets and constraints (2014) — link
- Quantum error correction for quantum memories (2013) — link
- Decision-Theoretic Planning with non-Markovian Rewards (2011) — link
- Maximum reward reinforcement learning: A non-cumulative reward criterion (2006) — link
- Learning to trade via direct reinforcement (2001) — link
- Policy Gradient Methods for Reinforcement Learning with Function Approximation (1999)
- Performance functions and reinforcement learning for trading systems and portfolios (1998) — link
- Structured Solution Methods for Non-Markovian Decision Processes (1997)
- Rewarding Behaviors (1996)
- Technical Note: Q-Learning (1992) — link
- Stable-Baselines3: Reliable Reinforcement Learning Implementations (2021)
- What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study (2021)
- Planning with General Objective Functions: Going Beyond Total Rewards (2020)
- Antonin Raffin (2020)
- Reinforcement learning - an introduction, 2nd Edition (2018)
- MUTUAL FUND PERFORMANCE* (2007) — link
- Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (2004) — link
- Code for quantum error correction code discovery
- Remmy Zen. Code for logical state preparation
- Code for ZX-diagrams
- Code for classical control, portfolio optimization, and peak environment
1 of 45 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.