mlsanityData to 14 Oct 2026

Feasible Adversarial Robust Reinforcement Learning for Underspecified Environments

arXiv 2207.09597 ↗First posted 19 Jul 2022cs.LG cs.AI cs.GTcited by 4

Not on the map. Why?

Abstract (arXiv)

Robust reinforcement learning (RL) considers the problem of learning policies that perform well in the worst case among a set of possible environment parameter values. In real-world environments, choosing the set of possible values for robust RL can be a difficult task. When that set is specified too narrowly, the agent will be left vulnerable to reasonable parameter values unaccounted for. When specified too broadly, the agent will be too cautious. In this paper, we propose Feasible Adversarial Robust RL (FARR), a novel problem formulation and objective for automatically determining the set of environment parameter values over which to be robust. FARR implicitly defines the set of feasible parameter values as those on which an agent could achieve a benchmark reward given enough training resources. By formulating this problem as a two-player zero-sum game, optimizing the FARR objective jointly produces an adversarial distribution over parameter values with feasible support and a policy robust over this feasible parameter set. We demonstrate that approximate Nash equilibria for this objective can be found using a variation of the PSRO algorithm. Furthermore, we show that an optimal agent trained with FARR is more robust to feasible adversarial parameter selection than with existing minimax, domain-randomization, and regret objectives in a parameterized gridworld and three MuJoCo control environments.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 34 of its 47 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

34 of 47 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  5. 2018Learning Dexterous In-Hand Manipulationcited by 687
  6. 2017Robust Adversarial Reinforcement Learningcited by 451
  7. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  8. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
  9. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  10. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  11. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  12. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
22 more
  1. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  2. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  3. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  4. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
  5. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  6. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  7. 2021Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robotscited by 116
  8. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 106
  9. 2018Deep Counterfactual Regret Minimizationcited by 103
  10. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 88
  11. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 72
  12. 2020Robust Reinforcement Learning using Adversarial Populationscited by 66
  13. 2021Policy Smoothing for Provably Robust Reinforcement Learningcited by 40
  14. 2021XDO: A Double Oracle Algorithm for Extensive-Form Gamescited by 34
  15. 2020Robustifying Reinforcement Learning Agents via Action Space Adversarial Trainingcited by 30
  16. 2019Automated curricula through setter-solver interactionscited by 26
  17. 2019How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?cited by 25
  18. 2019Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learningcited by 23
  19. 2019Distributionally Robust Partially Observable Markov Decision Process with Moment-based Ambiguitycited by 15
  20. 2022NeuPL: Neural Population Learningcited by 14
  21. 2022It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generationcited by 14
  22. 2022Self-Play PSRO: Toward Optimal Populations in Two-Player Zero-Sum Gamescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 0; 2025: 0; 2026: 1.

4 papers.

  1. 2022Illusory Attacks: Information-Theoretic Detectability Matters in Adversarial Attackscited by 12
  2. 2023JiangJun: Mastering Xiangqi by Tackling Non-Transitivity in Two-Player Zero-Sum Gamescited by 2
  3. 2023Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbationscited by 0
  4. 2026Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learningcited by 0