mlsanityData to 14 Oct 2026

Distributionally Robust Deep Q-Learning

arXiv 2505.19058 ↗First posted 25 May 2025cs.LG math.OC q-fin.PMcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We propose a novel distributionally robust Q-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case transition from a ball around a reference probability measure. To determine the optimal policy under the worst-case state transition, we solve the associated non-linear Bellman equation by dualising and regularising the Bellman operator with the Sinkhorn distance, which is then parameterised with deep neural networks. This approach allows us to modify the Deep Q-Network algorithm to optimise for the worst case state transition. We illustrate the tractability and effectiveness of our approach through several applications, including a portfolio optimisation task based on S&P~500 data. We also establish convergence guarantees for exact and approximate robust fitted Q-iteration, decompose the numerical RDQN error into interpretable components, and discuss extensions to compact continuous action sets.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 69 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

34 of 69 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Q-Learning under Finite Model Uncertainty5 Jul

Other references · 33

  1. 2021Convex optimizationcited by 2,515
  2. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  5. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  6. 2019Universal Approximation with Deep Narrow Networkscited by 200
  7. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  8. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 114
  9. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  10. 2022Robust Reinforcement Learning using Offline Datacited by 96
  11. 2016Playing FPS Games with Deep Reinforcement Learningcited by 95
  12. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
21 more
  1. 2019Sinkhorn Divergences for Unbalanced Optimal Transportcited by 55
  2. 2019Distributionally Robust Reinforcement Learningcited by 42
  3. 2021Sinkhorn Distributionally Robust Optimizationcited by 38
  4. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 38
  5. 2023A Finite Sample Complexity Bound for Distributionally Robust Q-learningcited by 36
  6. 2019On the Efficiency of Entropic Regularized Algorithms for Optimal Transportcited by 35
  7. 2018Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approachcited by 29
  8. 2023Sample Complexity of Variance-reduced Distributionally Robust Q-learningcited by 29
  9. 2021Stability of Entropic Optimal Transport and Schr"odinger Bridgescited by 26
  10. 2020On Linear Optimization over Wasserstein Ballscited by 24
  11. 2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
  12. 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 16
  13. 2020Distributionally Robust Markov Decision Processes and their Connection to Risk Measurescited by 14
  14. 2017Computational aspects of robust optimized certainty equivalents and option pricingcited by 7
  15. 2024Universal approximation results for neural networks with non-polynomial activation function over non-compact domainscited by 7
  16. 2023An Efficient Solution to s-Rectangular Robust Markov Decision Processescited by 6
  17. 2022Markov Decision Processes under Model Uncertaintycited by 3
  18. 2023Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularizationcited by 1
  19. 2023Neural networks can detect model-free static arbitrage strategiescited by 1
  20. 2024Robust SGLD algorithm for solving non-convex distributionally robust optimisation problemscited by 1
  21. 2024Non-concave stochastic optimal control in finite discrete time under model uncertaintycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Robust Q-learning for mean-field control under Wasserstein uncertainty in common noisecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximation
  2. 2024Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error
  3. 2025Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning
  4. 2024Zero-Sum Positional Differential Games as a Framework for Robust Reinforcement Learning: Deep Q-Learning Approach
  5. 2026Revisiting TD Target Aggregation under Uncertainty in Q-Learning
  6. 2026Deep Q-Learning on H"older Spaces
  7. 2024Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts
  8. 2025Deep neural networks can provably solve Bellman equations for Markov decision processes without the curse of dimensionality