Paper: arXiv 2505.19058

Authors: Chung I Lu, Julian Sester, Aijia Zhang

Abstract

We propose a novel distributionally robust \(Q\)-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case transition from a ball around a reference probability measure. To determine the optimal policy under the worst-case state transition, we solve the associated non-linear Bellman equation by dualising and regularising the Bellman operator with the Sinkhorn distance, which is then parameterized with deep neural networks. This approach allows us to modify the Deep Q-Network algorithm to optimise for the worst case state transition. We illustrate the tractability and effectiveness of our approach through several applications, including a portfolio optimisation task based on S&{“P”}~500 data.

Complexity vs Empirical Score

  • Math Complexity: 9.0/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematics including dualization of the Bellman operator and Sinkhorn distances for robust optimization, and it demonstrates effectiveness through a portfolio optimization task on S&P 500 data, indicating substantial empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal"] -->|"Develop DRL algorithm robust to model uncertainty"| B["Methodology: Distributionally Robust Q-Learning"]
  B --> C["Modeling Uncertainty"]
  C --> D["Solve Non-linear Bellman Equation"]
  D --> E["Optimization: Sinkhorn Distance + Dualization"]
  E --> F["Deep Q-Network Implementation"]
  F --> G["Experimental Application"]
  G --> H["Findings: Robustness & Performance"]
  H --> I["S&P 500 Portfolio Optimization"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a distributionally robust Q‑learning algorithm for continuous state spaces, using a Sinkhorn‑distance ambiguity set and a deep neural network implementation called Robust DQN (RDQN).

Source sentence
We propose a novel distributionally robust Q-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty.

Assumes 2

  • Assumptions 2.4 and 2.6 hold true model
    Source sentence
    Assume that Assumptions 2.4 and 2.6 hold true, then for δ ≥0 and ε > 0, the map
  • State space is compact other
    Source sentence
    We provide theorectical gaurantees for the existence of solutions when the state space is compact.

Data

Backtest sample: 1995-01 – 2023-12 · S&P 500 index · equities · daily

  • S&P 500 index data market-prices unknown
    3 Jan 1995 to 28 Dec 2023 · S&P 500 constituents
    Source sentence
    we generate 1,000 paths independent from the training dataset, each of length equal to the S&P 500 data period from 3 Jan 1995 to 28 Dec 2023.

Costs

Modeled Costs are part of the reported headline results.

0.05% transaction cost used in main experiments; also tests with zero cost and 0.25% during training.

Source sentence
This agent is trained with ε = 0.0025, δ = 0.0001 and a transaction cost of 0.05%.

Finds 4

Reported performance (Robust DQN (RDQN)): Sharpe 0.52 · max drawdown 34.7%

  • RDQN outperforms DQN on all average risk‑adjusted metrics despite lower absolute wealth. performance
    Source sentence
    we observe see a reversal of fortune for the RDQN agent where it is now outperforming the DQN agent on all average risk-adjusted metrics.
  • With zero transaction cost both agents accumulate significantly more wealth, and RDQN achieves higher Sharpe and Sortino ratios than DQN and the S&P 500 index. performance
    Source sentence
    ... the Sharpe and Sortino ratios are also significantly higher with the RDQN agents outperforming both the DQN agent and the S&P 500 index.
  • In the simulator, the DQN agent delivers significantly better wealth than the average simulated path. performance
    Source sentence
    It is clear from the wealth metric, that the DQN agent delivers significantly better results than the average simulated path
  • Overall, RDQN shows better adaptation to distributional shift than DQN across all result sets. performance
    Source sentence
    Overall, we see better adaptation to the distributional shift with the RDQN agent compared to the DQN agent consistently across all three sets of results

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 50 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 1 paper in the archive (1 by independent authors)

Code

https://github.com/luchungi/Sinkhorn_RDQN — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (84)

  1. Playing FPS games with deep reinforcement learning (2017)
  2. Asynchronous methods for deep reinforcement learning (2016)
  3. Deep reinforcement learning with double Q -learning (2016)
  4. Addressing function approximation error in actor-critic methods (2018)
  5. Optimally combining sampling techniques for Monte Carlo rendering (1995)
  6. Scrambling Sobol’and Niederreiter–Xing Points (1998)
  7. Continuous control with deep reinforcement learning (2016)
  8. Twice regularized Markov decision processes: The equivalence between robustness and regularization (2023) — link
and 76 more
  1. Robustness in Sequential Decision Making under Evolving Uncertainty: Evidence from High-Frequency Market Making (2026) — link
  2. Kullback-Leibler divergence constrained distributionally robust optimization (2013)
  3. Wasserstein distributionally robust optimization: Theory and applications in machine learning (2019)
  4. Optimal control of execution costs (1998) — link
  5. Human-level control through deep reinforcement learning (2015) — link
  6. Distributionally Robust Deep Q-Learning (2025) — link
  7. Rectangularity and duality of distributionally robust Markov decision processes (2023) — link
  8. Optimal transport: old and new (2009)
  9. Introduction to entropic optimal transport (2021)
  10. Principles of mathematical analysis (1964)
  11. Robust Q -learning algorithm for Markov decision processes under Wasserstein uncertainty (2024)
  12. Generative model for financial time series trained with MMD using a signature kernel — → 2407.19848
  13. Learning to utilize shaping rewards: A new approach of reward shaping (2020)
  14. Universal approximation results for neural networks with non-polynomial activation function over non-compact domains (2024) — link
  15. Dota 2 with large scale deep reinforcement learning (2019) — link
  16. Probability essentials (2012)
  17. Probability theory: a comprehensive course (2013)
  18. Double Q -learning (2010)
  19. On the theory of dynamic programming (1952)
  20. Q-learning (1992) — link
  21. Reinforcement learning: An introduction (2018)
  22. Computational aspects of robust optimized certainty equivalents and option pricing (2020)
  23. Distributionally robust Markov decision processes (2010)
  24. Sample complexity of variance-reduced distributionally robust Q -learning (2024)
  25. Infinite dimensional analysis: a hitchhiker’s guide (2006)
  26. Sinkhorn divergences for unbalanced optimal transport (2019) — link
  27. Nonparametric adaptive robust control under model uncertainty (2023)
  28. Robust Markov decision processes (2013)
  29. Improved sample complexity bounds for distributionally robust reinforcement learning (2023)
  30. Neural networks can detect model-free static arbitrage strategies — → 2306.16422
  31. Universal approximation with deep narrow networks (2020)
  32. Distributionally robust reinforcement learning (2019) — link
  33. A convex optimization approach to distributionally robust Markov decision processes with Wasserstein distance (2017)
  34. On linear optimization over Wasserstein balls (2022)
  35. Robust control of Markov decision processes with uncertain transition matrices (2005)
  36. Robust dynamic programming (2005)
  37. An Efficient Solution to s-Rectangular Robust Markov Decision Processes (2023) — link
  38. Non-concave distributionally robust stochastic control in a discrete time finite horizon setting — → 2404.05230
  39. Policy Gradient in Robust MDP s with Global Convergence Guarantee (2023) — link
  40. Interpolating between optimal transport and mmd using Sinkhorn divergences (2019)
  41. Convex optimization (2004)
  42. Policy Gradient Method For Robust Reinforcement Learning (2022) — link
  43. Markov decision processes under model uncertainty (2023)
  44. Distributionally robust Markov decision processes and their connection to risk measures (2022)
  45. Computational Optimal Transport: With Applications to Data Science (2019)
  46. Robust Markov Decision Processes: Beyond Rectangularity (2023) — link
  47. Policy gradient algorithms for robust MDP s with non-rectangular uncertainty sets (2023) — link
  48. Distributionally robust optimization: A review (2019) — link
  49. Distributionally robust convex optimization (2014)
  50. Robust MDPs with k-Rectangular Uncertainty (2016) — link
  51. A relationship between arbitrary positive matrices and doubly stochastic matrices (1964)
  52. Natural actor-critic for robust reinforcement learning with function approximation (2023)
  53. Online robust reinforcement learning with model uncertainty (2021)
  54. Mathematical foundations of distributionally robust multistage optimization (2021)
  55. Investigating the parameters of the beta distribution (2023)
  56. Sinkhorn distributionally robust optimization (2021) — link
  57. Maximum entropy RL (provably) solves some robust RL problems (2021) — link
  58. Distributionally Robust Q -Learning (2022)
  59. On the efficiency of entropic regularized algorithms for optimal transport (2022)
  60. Robust Q -Learning for finite ambiguity sets (2024) — link
  61. Robust reinforcement learning using offline data (2022)
  62. A theoretical analysis of deep Q -learning (2020)
  63. Distributionally robust model-based reinforcement learning with large state spaces (2024)
  64. Multi-stage distributionally robust stochastic dual dynamic programming to multi-period economic dispatch with virtual energy storage (2021)
  65. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks (1990)
  66. Sinkhorn Distances: Lightspeed Computation of Optimal Transport (2013) — link
  67. Universal approximation using feedforward neural networks: A survey of some existing methods, and some new results (1998)
  68. A finite sample complexity bound for distributionally robust Q -learning (2023)
  69. Wasserstein distributionally robust stochastic control: A data-driven approach (2020)
  70. Stability of entropic optimal transport and Schrödinger bridges (2022)
  71. Approximation theory of the MLP model in neural networks (1999)
  72. Deep reinforcement learning: From Q -learning to deep Q -learning (2017)
  73. Robust SGLD algorithm for solving non-convex distributionally robust optimisation problems — → 2403.09532
  74. Structural estimation of Markov decision processes (1994)
  75. Sample complexity of Sinkhorn divergences (2019)
  76. Empirical properties of asset returns: stylized facts and statistical issues (2001) — link

4 of 84 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.