mlsanityData to 7 Oct 2026

Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variables

arXiv 2210.12262 ↗First posted 21 Oct 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

One key challenge for multi-task Reinforcement learning (RL) in practice is the absence of task indicators. Robust RL has been applied to deal with task ambiguity, but may result in over-conservative policies. To balance the worst-case (robustness) and average performance, we propose Group Distributionally Robust Markov Decision Process (GDR-MDP), a flexible hierarchical MDP formulation that encodes task groups via a latent mixture model. GDR-MDP identifies the optimal policy that maximizes the expected return under the worst-possible qualified belief over task groups within an ambiguity set. We rigorously show that GDR-MDP's hierarchical structure improves distributional robustness by adding regularization to the worst possible outcomes. We then develop deep RL algorithms for GDR-MDP for both value-based and policy-based RL methods. Extensive experiments on Box2D control tasks, MuJoCo benchmarks, and Google football platforms show that our algorithms outperform classic robust training algorithms across diverse environments in terms of robustness under belief uncertainties. Demos are available on our project page (urlhttps://sites.google.com/view/gdr-rl/home).

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 40 of its 74 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

40 of 74 references.

References · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  7. 2017Robust Adversarial Reinforcement Learningcited by 439
  8. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  9. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  10. 2017Adversarial Attacks on Neural Network Policiescited by 314
  11. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  12. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
28 more
  1. 2017Learning with Opponent-Learning Awarenesscited by 218
  2. 2015Contextual Markov Decision Processescited by 213
  3. 2019Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learningcited by 169
  4. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  5. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  6. 2019Distributionally Robust Language Modelingcited by 142
  7. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  8. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 129
  9. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 69
  10. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 68
  11. 2013Sample Complexity of Multi-task Reinforcement Learningcited by 58
  12. 2017QMDP-Net: Deep Learning for Planning under Partial Observabilitycited by 51
  13. 2019Wasserstein Robust Reinforcement Learningcited by 50
  14. 2019Distributionally Robust Reinforcement Learningcited by 40
  15. 2019Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended versioncited by 36
  16. 2021Hierarchical Bayesian Banditscited by 33
  17. 2015Distributionally Robust Counterpart in Markov Decision Processescited by 32
  18. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  19. 2022Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Trainingcited by 26
  20. 2018Robust Markov Decision Process: Beyond Rectangularitycited by 24
  21. 2020Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processescited by 23
  22. 2020First-Order Methods for Wasserstein Distributionally Robust MDPcited by 16
  23. 2022Robust Policy Learning over Multiple Uncertainty Setscited by 16
  24. 2019Distributionally Robust Partially Observable Markov Decision Process with Moment-based Ambiguitycited by 15
  25. 2018Bayesian Policy Optimization for Model Uncertaintycited by 12
  26. 2019Robust and Adaptive Planning under Model Uncertaintycited by 10
  27. 2020FormulaZero: Distributionally Robust Online Adaptation via Offline Population Synthesiscited by 10
  28. 2013Time (in)consistency of multistage distributionally robust inventory models with moment constraintscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1; 2025: 1.

6 papers.

  1. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  2. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  3. 2024CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectoriescited by 11
  4. 2023Wasserstein Distributionally Robust Control Barrier Function using Conditional Value-at-Risk with Differentiable Convex Programmingcited by 2
  5. 2025Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mappingcited by 2
  6. 2023Distributionally Safe Reinforcement Learning under Model Uncertainty: A Single-Level Approach by Differentiable Convex Programmingcited by 1