mlsanityData to 7 Oct 2026

Robust Reinforcement Learning for Continuous Control with Model Misspecification

arXiv 1906.07516 ↗First posted 18 Jun 2019cs.LG cs.AI stat.MLcited by 76

Not on the map. Why?

Abstract (arXiv)

We provide a framework for incorporating robustness -- to perturbations in the transition dynamics which we refer to as model misspecification -- into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve this by learning a policy that optimizes for a worst case expected return objective and derive a corresponding robust entropy-regularized Bellman contraction operator. In addition, we introduce a less conservative, soft-robust, entropy-regularized objective with a corresponding Bellman operator. We show that both, robust and soft-robust policies, outperform their non-robust counterparts in nine Mujoco domains with environment perturbations. In addition, we show improved robust performance on a high-dimensional, simulated, dexterous robotic hand. Finally, we present multiple investigative experiments that provide a deeper insight into the robustness framework. This includes an adaptation to another continuous control RL algorithm as well as learning the uncertainty set from offline data. Performance videos can be found online at https://sites.google.com/view/robust-rl.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 16; 2022: 18; 2023: 17; 2024: 9; 2025: 10; 2026: 5.

76 papers.

Sort:

By area:

  1. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  2. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
  3. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  4. 2022Robust Reinforcement Learning using Offline Datacited by 95
  5. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  6. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  7. 2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 42
  8. 2021Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challengescited by 40
  9. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  10. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  11. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  12. 2022Robust Imitation Learning against Variations in Environment Dynamicscited by 26
38 more
  1. 2024Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximationcited by 25
  2. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  3. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 24
  4. 2025RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbationscited by 23
  5. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Beliefcited by 22
  6. 2021Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimizationcited by 18
  7. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 18
  8. 2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
  9. 2022Robust Policy Learning over Multiple Uncertainty Setscited by 16
  10. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
  11. 2021Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexitycited by 15
  12. 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
  13. 2023Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learningcited by 12
  14. 2022Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecificationcited by 11
  15. 2022Robust Learning from Observation with Model Misspecificationcited by 9
  16. 2022Provable Sim-to-real Transfer in Continuous Domain with Partial Observationscited by 9
  17. 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guaranteescited by 9
  18. 2024Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Spacecited by 9
  19. 2022Sparse Adversarial Attack in Multi-agent Reinforcement Learningcited by 8
  20. 2022Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularizationcited by 7
  21. 2025Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learningcited by 6
  22. 2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
  23. 2021Robust Entropy-regularized Markov Decision Processescited by 4
  24. 2022RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Riskcited by 4
  25. 2025Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approachcited by 4
  26. 2022FI-ODE: Certifiably Robust Forward Invariance in Neural ODEscited by 3
  27. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  28. 2021Robust Reinforcement Learning under model misspecificationcited by 2
  29. 2021On Assessing The Safety of Reinforcement Learning algorithms Using Formal Methodscited by 2
  30. 2022Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learningcited by 2
  31. 2023Certifiably Robust Reinforcement Learning through Model-Based Abstract Interpretationcited by 2
  32. 2023Online Nonstochastic Model-Free Reinforcement Learningcited by 2
  33. 2023Robust Reinforcement Learning through Efficient Adversarial Herdingcited by 2
  34. 2023Wasserstein Distributionally Robust Control Barrier Function using Conditional Value-at-Risk with Differentiable Convex Programmingcited by 2
  35. 2024Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Datacited by 2
  36. 2024GRAM: Generalization in Deep RL with a Robust Adaptation Modulecited by 2
  37. 2025Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysiscited by 2
  38. 2025Improving Controller Generalization with Dimensionless Markov Decision Processescited by 2

50 most cited of 76 All 76 (JSON)