mlsanityData to 7 Oct 2026

Distributionally Robust Model-based Reinforcement Learning with Large State Spaces

arXiv 2309.02236 ↗First posted 5 Sep 2023cs.LG cs.AI stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

Three major challenges in reinforcement learning are the complex dynamical systems with large state spaces, the costly data acquisition processes, and the deviation of real-world dynamics from the training environment deployment. To overcome these issues, we study distributionally robust Markov decision processes with continuous state spaces under the widely used Kullback-Leibler, chi-square, and total variation uncertainty sets. We propose a model-based approach that utilizes Gaussian Processes and the maximum variance reduction algorithm to efficiently learn multi-output nominal transition dynamics, leveraging access to a generative model (i.e., simulator). We further demonstrate the statistical sample complexity of the proposed method for different uncertainty sets. These complexity bounds are independent of the number of states and extend beyond linear dynamics, ensuring the effectiveness of our approach in identifying near-optimal distributionally-robust policies. The proposed method can be further combined with other model-free distributionally robust reinforcement learning methods to obtain a near-optimal robust policy. Experimental results demonstrate the robustness of our algorithm to distributional shifts and its superior performance in terms of the number of samples needed.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 52 of its 98 references that are in the map. First posted on arXiv 5 Sep 2023.

Builds on

52 of 98 references.

References · 52

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016OpenAI Gymcited by 1,684
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  4. 2018Learning Dexterous In-Hand Manipulationcited by 643
  5. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  6. 2017Robust Adversarial Reinforcement Learningcited by 439
  7. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  8. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  9. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  10. 2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 188
  11. 2019Model-Based Reinforcement Learning with a Generative Model is Minimax Optimalcited by 124
  12. 2019Learning with Good Feature Representations in Bandits and in RL with a Generative Modelcited by 124
38 more
  1. 2020Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Modelcited by 108
  2. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  3. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  4. 2022Robust Reinforcement Learning using Offline Datacited by 95
  5. 2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 90
  6. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  7. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  8. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  9. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  10. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  11. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 76
  12. 2018Adversarially Robust Optimization with Gaussian Processescited by 61
  13. 2020Distributionally Robust Bayesian Optimizationcited by 60
  14. 2016Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Modelcited by 53
  15. 2017Reinforcement Learning under Model Mismatchcited by 52
  16. 2021Optimal Order Simple Regret for Gaussian Process Banditscited by 52
  17. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  18. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
  19. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  20. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  21. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 37
  22. 2023A Finite Sample Complexity Bound for Distributionally Robust Q-learningcited by 35
  23. 2015Distributionally Robust Counterpart in Markov Decision Processescited by 32
  24. 2019A Bayesian Approach to Robust Reinforcement Learningcited by 32
  25. 2019On Online Learning in Kernelized Markov Decision Processescited by 32
  26. 2023Sample Complexity of Variance-reduced Distributionally Robust Q-learningcited by 28
  27. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  28. 2021Efficient Local Planning with Linear Function Approximationcited by 21
  29. 2023Towards Minimax Optimality of Model-based Robust Reinforcement Learningcited by 21
  30. 2019Sample Complexity of Sample Average Approximation for Conditional Stochastic Optimizationcited by 19
  31. 2020Efficient Planning in Large MDPs with Weak Linear Function Approximationcited by 17
  32. 2017Mutual Alignment Transfer Learningcited by 16
  33. 2021An Experimental Design Perspective on Model-Based Reinforcement Learningcited by 16
  34. 2020Distributionally Robust Bayesian Quadrature Optimizationcited by 15
  35. 2023Single-Trajectory Distributionally Robust Reinforcement Learningcited by 15
  36. 2021Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learningcited by 14
  37. 2022Near-optimal Policy Identification in Active Reinforcement Learningcited by 7
  38. 2022Generalization Bounds of Nonconvex-(Strongly)-Concave Stochastic Minimax Optimizationcited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 2.

17 papers.

Sort:

By area:

  1. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 24
  2. 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processescited by 12
  3. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
  4. 2025Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learningcited by 6
  5. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversioncited by 3
  6. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  7. 2025Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mappingcited by 2
  8. 2025Towards Optimal Offline Reinforcement Learningcited by 2
  9. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
  10. 2025Distributionally Robust Deep Q-Learningcited by 1
  11. 2025Distributionally Robust Optimization with Adversarial Data Contaminationcited by 1
  12. 2025Distributionally Robust Self Paced Curriculum Reinforcement Learningcited by 1
5 more
  1. 2025Perturbation-mitigated USV Navigation with Distributionally Robust Reinforcement Learningcited by 1
  2. 2025Online Robust Reinforcement Learning with General Function Approximationcited by 1
  3. 2025Tail-Safe Hedging: Explainable Risk-Sensitive Reinforcement Learning with a White-Box CBF--QP Safety Layer in Arbitrage-Free Marketscited by 0
  4. 2026Partial GFlowNet: Accelerating Convergence in Large State Spaces via Strategic Partitioningcited by 0
  5. 2026Data-driven robust Markov decision processes on Borel spaces: performance guarantees via an axiomatic approachcited by 0