mlsanityData to 7 Oct 2026

Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning

arXiv 2301.12593 ↗First posted 30 Jan 2023cs.LG cs.AI stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

Many real-world domains require safe decision making in uncertain environments. In this work, we introduce a deep reinforcement learning framework for approaching this important problem. We consider a distribution over transition models, and apply a risk-averse perspective towards model uncertainty through the use of coherent distortion risk measures. We provide robustness guarantees for this framework by showing it is equivalent to a specific class of distributionally robust safe reinforcement learning problems. Unlike existing approaches to robustness in deep reinforcement learning, however, our formulation does not involve minimax optimization. This leads to an efficient, model-free implementation of our approach that only requires standard data collection from a single training environment. In experiments on continuous control tasks with safety constraints, we demonstrate that our framework produces robust performance and safety at deployment time across a range of perturbed test environments.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 41 of its 67 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

41 of 67 references.

References · 41

  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  2. 2017Constrained Policy Optimizationcited by 928
  3. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  6. 2018Learning Dexterous In-Hand Manipulationcited by 643
  7. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  8. 2016Deep Exploration via Bootstrapped DQNcited by 518
  9. 2017Robust Adversarial Reinforcement Learningcited by 439
  10. 2018Reward Constrained Policy Optimizationcited by 397
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2018Maximum a Posteriori Policy Optimisationcited by 318
29 more
  1. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  2. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
  3. 2018Safe Exploration in Continuous Action Spacescited by 237
  4. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  5. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  6. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  7. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  8. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  9. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  10. 2013Risk-sensitive Reinforcement Learningcited by 129
  11. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  12. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  13. 2020Conservative Safety Critics for Explorationcited by 107
  14. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 76
  15. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 75
  16. 2015Policy Gradient for Coherent Risk Measurescited by 74
  17. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  18. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  19. 2020Robust Reinforcement Learning using Adversarial Populationscited by 64
  20. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  21. 2017How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Roboticscited by 57
  22. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  23. 2020A Distributional View on Multi-Objective Policy Optimizationcited by 51
  24. 2021Distributionally Robust Learningcited by 44
  25. 2020Distributional Robustness and Regularization in Reinforcement Learningcited by 39
  26. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  27. 2015Distributionally Robust Counterpart in Markov Decision Processescited by 32
  28. 2020Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecificationcited by 16
  29. 2022Domain Knowledge-Based Automated Analog Circuit Design with Deep Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 3; 2026: 3.

12 papers.

  1. 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
  2. 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guaranteescited by 9
  3. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
  4. 2026Safe Exploration via Policy Priorscited by 6
  5. 2024Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Modelcited by 3
  6. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  7. 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learningcited by 2
  8. 2024GRAM: Generalization in Deep RL with a Robust Adaptation Modulecited by 2
  9. 2023Distributionally Safe Reinforcement Learning under Model Uncertainty: A Single-Level Approach by Differentiable Convex Programmingcited by 1
  10. 2025TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labelscited by 0
  11. 2026Risk-Aware Human-in-the-Loop Framework with Adaptive Intrusion Response for Autonomous Vehiclescited by 0
  12. 2026Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertaintycited by 0