Paper: arXiv 2505.19058
Authors: Chung I Lu, Julian Sester, Aijia Zhang
Abstract
We propose a novel distributionally robust \(Q\)-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case transition from a ball around a reference probability measure. To determine the optimal policy under the worst-case state transition, we solve the associated non-linear Bellman equation by dualising and regularising the Bellman operator with the Sinkhorn distance, which is then parameterized with deep neural networks. This approach allows us to modify the Deep Q-Network algorithm to optimise for the worst case state transition. We illustrate the tractability and effectiveness of our approach through several applications, including a portfolio optimisation task based on S&{“P”}~500 data.
Complexity vs Empirical Score
- Math Complexity: 9.0/10
- Empirical Rigor: 7.5/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper employs advanced mathematics including dualization of the Bellman operator and Sinkhorn distances for robust optimization, and it demonstrates effectiveness through a portfolio optimization task on S&P 500 data, indicating substantial empirical rigor.
Research Flowchart
flowchart TD A["Research Goal"] -->|"Develop DRL algorithm robust to model uncertainty"| B["Methodology: Distributionally Robust Q-Learning"] B --> C["Modeling Uncertainty"] C --> D["Solve Non-linear Bellman Equation"] D --> E["Optimization: Sinkhorn Distance + Dualization"] E --> F["Deep Q-Network Implementation"] F --> G["Experimental Application"] G --> H["Findings: Robustness & Performance"] H --> I["S&P 500 Portfolio Optimization"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a distributionally robust Q‑learning algorithm for continuous state spaces, using a Sinkhorn‑distance ambiguity set and a deep neural network implementation called Robust DQN (RDQN).
Source sentence
We propose a novel distributionally robust Q-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty.
Assumes 2
- Assumptions 2.4 and 2.6 hold true model
Source sentence
Assume that Assumptions 2.4 and 2.6 hold true, then for δ ≥0 and ε > 0, the map
- State space is compact other
Source sentence
We provide theorectical gaurantees for the existence of solutions when the state space is compact.
Data
1995-01 – 2023-12 · S&P 500 index · equities · daily
- S&P 500 index data market-prices unknown
Source sentence
we generate 1,000 paths independent from the training dataset, each of length equal to the S&P 500 data period from 3 Jan 1995 to 28 Dec 2023.
Costs
Modeled
0.05% transaction cost used in main experiments; also tests with zero cost and 0.25% during training.
Source sentence
This agent is trained with ε = 0.0025, δ = 0.0001 and a transaction cost of 0.05%.
Finds 4
Sharpe 0.52 · max drawdown 34.7%
- RDQN outperforms DQN on all average risk‑adjusted metrics despite lower absolute wealth. performance
Source sentence
we observe see a reversal of fortune for the RDQN agent where it is now outperforming the DQN agent on all average risk-adjusted metrics.
- With zero transaction cost both agents accumulate significantly more wealth, and RDQN achieves higher Sharpe and Sortino ratios than DQN and the S&P 500 index. performance
Source sentence
... the Sharpe and Sortino ratios are also significantly higher with the RDQN agents outperforming both the DQN agent and the S&P 500 index.
- In the simulator, the DQN agent delivers significantly better wealth than the average simulated path. performance
Source sentence
It is clear from the wealth metric, that the DQN agent delivers significantly better results than the average simulated path
- Overall, RDQN shows better adaptation to distributional shift than DQN across all result sets. performance
Source sentence
Overall, we see better adaptation to the distributional shift with the RDQN agent compared to the DQN agent consistently across all three sets of results
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 50 related papers). Coverage is partial, so this is neither replication nor refutation.
1 paper in the archive
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (84)
- Playing FPS games with deep reinforcement learning (2017)
- Asynchronous methods for deep reinforcement learning (2016)
- Deep reinforcement learning with double Q -learning (2016)
- Addressing function approximation error in actor-critic methods (2018)
- Optimally combining sampling techniques for Monte Carlo rendering (1995)
- Scrambling Sobol’and Niederreiter–Xing Points (1998)
- Continuous control with deep reinforcement learning (2016)
- Twice regularized Markov decision processes: The equivalence between robustness and regularization (2023) — link
and 76 more
- Robustness in Sequential Decision Making under Evolving Uncertainty: Evidence from High-Frequency Market Making (2026) — link
- Kullback-Leibler divergence constrained distributionally robust optimization (2013)
- Wasserstein distributionally robust optimization: Theory and applications in machine learning (2019)
- Optimal control of execution costs (1998) — link
- Human-level control through deep reinforcement learning (2015) — link
- Distributionally Robust Deep Q-Learning (2025) — link
- Rectangularity and duality of distributionally robust Markov decision processes (2023) — link
- Optimal transport: old and new (2009)
- Introduction to entropic optimal transport (2021)
- Principles of mathematical analysis (1964)
- Robust Q -learning algorithm for Markov decision processes under Wasserstein uncertainty (2024)
- Generative model for financial time series trained with MMD using a signature kernel — → 2407.19848
- Learning to utilize shaping rewards: A new approach of reward shaping (2020)
- Universal approximation results for neural networks with non-polynomial activation function over non-compact domains (2024) — link
- Dota 2 with large scale deep reinforcement learning (2019) — link
- Probability essentials (2012)
- Probability theory: a comprehensive course (2013)
- Double Q -learning (2010)
- On the theory of dynamic programming (1952)
- Q-learning (1992) — link
- Reinforcement learning: An introduction (2018)
- Computational aspects of robust optimized certainty equivalents and option pricing (2020)
- Distributionally robust Markov decision processes (2010)
- Sample complexity of variance-reduced distributionally robust Q -learning (2024)
- Infinite dimensional analysis: a hitchhiker’s guide (2006)
- Sinkhorn divergences for unbalanced optimal transport (2019) — link
- Nonparametric adaptive robust control under model uncertainty (2023)
- Robust Markov decision processes (2013)
- Improved sample complexity bounds for distributionally robust reinforcement learning (2023)
- Neural networks can detect model-free static arbitrage strategies — → 2306.16422
- Universal approximation with deep narrow networks (2020)
- Distributionally robust reinforcement learning (2019) — link
- A convex optimization approach to distributionally robust Markov decision processes with Wasserstein distance (2017)
- On linear optimization over Wasserstein balls (2022)
- Robust control of Markov decision processes with uncertain transition matrices (2005)
- Robust dynamic programming (2005)
- An Efficient Solution to s-Rectangular Robust Markov Decision Processes (2023) — link
- Non-concave distributionally robust stochastic control in a discrete time finite horizon setting — → 2404.05230
- Policy Gradient in Robust MDP s with Global Convergence Guarantee (2023) — link
- Interpolating between optimal transport and mmd using Sinkhorn divergences (2019)
- Convex optimization (2004)
- Policy Gradient Method For Robust Reinforcement Learning (2022) — link
- Markov decision processes under model uncertainty (2023)
- Distributionally robust Markov decision processes and their connection to risk measures (2022)
- Computational Optimal Transport: With Applications to Data Science (2019)
- Robust Markov Decision Processes: Beyond Rectangularity (2023) — link
- Policy gradient algorithms for robust MDP s with non-rectangular uncertainty sets (2023) — link
- Distributionally robust optimization: A review (2019) — link
- Distributionally robust convex optimization (2014)
- Robust MDPs with k-Rectangular Uncertainty (2016) — link
- A relationship between arbitrary positive matrices and doubly stochastic matrices (1964)
- Natural actor-critic for robust reinforcement learning with function approximation (2023)
- Online robust reinforcement learning with model uncertainty (2021)
- Mathematical foundations of distributionally robust multistage optimization (2021)
- Investigating the parameters of the beta distribution (2023)
- Sinkhorn distributionally robust optimization (2021) — link
- Maximum entropy RL (provably) solves some robust RL problems (2021) — link
- Distributionally Robust Q -Learning (2022)
- On the efficiency of entropic regularized algorithms for optimal transport (2022)
- Robust Q -Learning for finite ambiguity sets (2024) — link
- Robust reinforcement learning using offline data (2022)
- A theoretical analysis of deep Q -learning (2020)
- Distributionally robust model-based reinforcement learning with large state spaces (2024)
- Multi-stage distributionally robust stochastic dual dynamic programming to multi-period economic dispatch with virtual energy storage (2021)
- Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks (1990)
- Sinkhorn Distances: Lightspeed Computation of Optimal Transport (2013) — link
- Universal approximation using feedforward neural networks: A survey of some existing methods, and some new results (1998)
- A finite sample complexity bound for distributionally robust Q -learning (2023)
- Wasserstein distributionally robust stochastic control: A data-driven approach (2020)
- Stability of entropic optimal transport and Schrödinger bridges (2022)
- Approximation theory of the MLP model in neural networks (1999)
- Deep reinforcement learning: From Q -learning to deep Q -learning (2017)
- Robust SGLD algorithm for solving non-convex distributionally robust optimisation problems — → 2403.09532
- Structural estimation of Markov decision processes (1994)
- Sample complexity of Sinkhorn divergences (2019)
- Empirical properties of asset returns: stylized facts and statistical issues (2001) — link
4 of 84 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.