Robust Reinforcement Learning through Efficient Adversarial Herding
Not on the map. Why?
Abstract (arXiv)
Although reinforcement learning (RL) is considered the gold standard for policy design, it may not always provide a robust solution in various scenarios. This can result in severe performance degradation when the environment is exposed to potential disturbances. Adversarial training using a two-player max-min game has been proven effective in enhancing the robustness of RL agents. In this work, we extend the two-player game by introducing an adversarial herd, which involves a group of adversaries, in order to address (textiti) the difficulty of the inner optimization problem, and (textitii) the potential over pessimism caused by the selection of a candidate adversary set that may include unlikely scenarios. We first prove that adversarial herds can efficiently approximate the inner optimization problem. Then we address the second issue by replacing the worst-case performance in the inner optimization with the average performance over the worst-k adversaries. We evaluate the proposed method on multiple MuJoCo environments. Experimental results demonstrate that our approach consistently generates more robust policies.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 12 Jun 2023.
Builds on
30 of 37 references.
References · 30
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Trust Region Policy Optimizationcited by 2,580
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
- 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
- 2019Model-Based Reinforcement Learning for Ataricited by 502
- 2017Robust Adversarial Reinforcement Learningcited by 451
- 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
18 more
- 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
- 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
- 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
- 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
- 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
- 2019Active Domain Randomizationcited by 170
- 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
- 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
- 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 78
- 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
- 2020Robust Reinforcement Learning using Adversarial Populationscited by 66
- 2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
- 2020Model-based Adversarial Meta-Reinforcement Learningcited by 33
- 2021Discovering a set of policies for the worst case rewardcited by 21
- 2021Safe Reinforcement Learning Using Robust Action Governorcited by 14
- 2020Model-Based Generalization Under Parameter Uncertainty Using Path Integral Controlcited by 7
- 2021Gradient Importance Learning for Incomplete Observationscited by 7
- 2019Adversary A3C for Robust Reinforcement Learningcited by 6
Built on by
Citing papers per year: 2024: 2.
2 papers.