mlsanityData to 7 Oct 2026

Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs

arXiv 1902.07605 ↗First posted 20 Feb 2019cs.LG stat.MLcited by 37

Not on the map. Why?

Abstract (arXiv)

Robust MDPs (RMDPs) can be used to compute policies with provable worst-case guarantees in reinforcement learning. The quality and robustness of an RMDP solution are determined by the ambiguity set---the set of plausible transition probabilities---which is usually constructed as a multi-dimensional confidence region. Existing methods construct ambiguity sets as confidence regions using concentration inequalities which leads to overly conservative solutions. This paper proposes a new paradigm that can achieve better solutions with the same robustness guarantees without using confidence regions as ambiguity sets. To incorporate prior knowledge, our algorithms optimize the size and position of ambiguity sets using Bayesian inference. Our theoretical analysis shows the safety of the proposed method, and the empirical results demonstrate its practical promise.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 7; 2022: 7; 2023: 11; 2024: 6; 2025: 3; 2026: 1.

37 papers.

Sort:

By area:

  1. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  2. 2022Robust Reinforcement Learning using Offline Datacited by 95
  3. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  4. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  5. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  6. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 35
  7. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
  8. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  9. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  10. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Beliefcited by 22
  11. 2023Replicable Reinforcement Learningcited by 21
  12. 2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
25 more
  1. 2022Robust Anytime Learning of Markov Decision Processescited by 16
  2. 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
  3. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
  4. 2023Decision-Making Under Uncertainty: Beyond Probabilitiescited by 8
  5. 2021Explicit Explore, Exploit, or Escape (E4): near-optimal safety-constrained reinforcement learning in polynomial timecited by 6
  6. 2023Learning the Uncertainty Sets for Control Dynamics via Set Membership: A Non-Asymptotic Analysiscited by 6
  7. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 6
  8. 2023Minimax-Bayes Reinforcement Learningcited by 5
  9. 2022RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Riskcited by 4
  10. 2021Learning Under Adversarial and Interventional Shiftscited by 3
  11. 2021Lyapunov Robust Constrained-MDPs: Soft-Constrained Robustly Stable Policy Optimization under Model Uncertaintycited by 3
  12. 2023Robust Lagrangian and Adversarial Policy Gradient for Robust Constrained Markov Decision Processescited by 2
  13. 2024Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Datacited by 2
  14. 2024Solving Multi-Model MDPs by Coordinate Ascent and Dynamic Programmingcited by 2
  15. 2021An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processescited by 0
  16. 2022Towards Robust Off-Policy Evaluation via Human Inputscited by 0
  17. 2023On Practical Robust Reinforcement Learning: Practical Uncertainty Set and Double-Agent Algorithmcited by 0
  18. 2023Bayesian Regret Minimization in Offline Banditscited by 0
  19. 2023Where2Start: Leveraging initial States for Robust and Sample-Efficient Reinforcement Learningcited by 0
  20. 2024Offline Risk-sensitive RL with Partial Observability to Enhance Performance in Human-Robot Teamingcited by 0
  21. 2024Percentile Criterion Optimization in Offline Reinforcement Learningcited by 0
  22. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processescited by 0
  23. 2025Smart Exploration in Reinforcement Learning using Bounded Uncertainty Modelscited by 0
  24. 2025Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functionscited by 0
  25. 2026Quantifying Risk Under Evolving Uncertainty: Belief-Dependent Robustness for Safe Sequential Decision Makingcited by 0