mlsanityData to 7 Oct 2026

Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training

arXiv 2202.09514 ↗First posted 19 Feb 2022cs.LG cs.AI cs.GTcited by 26

Not on the map. Why?

Abstract (arXiv)

Robust Reinforcement Learning (RL) focuses on improving performances under model errors or adversarial attacks, which facilitates the real-life deployment of RL agents. Robust Adversarial Reinforcement Learning (RARL) is one of the most popular frameworks for robust RL. However, most of the existing literature models RARL as a zero-sum simultaneous game with Nash equilibrium as the solution concept, which could overlook the sequential nature of RL deployments, produce overly conservative agents, and induce training instability. In this paper, we introduce a novel hierarchical formulation of robust RL - a general-sum Stackelberg game model called RRL-Stack - to formalize the sequential nature and provide extra flexibility for robust training. We develop the Stackelberg Policy Gradient algorithm to solve RRL-Stack, leveraging the Stackelberg learning dynamics by considering the adversary's response. Our method generates challenging yet solvable adversarial environments which benefit RL agents' robust learning. Our algorithm demonstrates better training stability and robustness against different testing conditions in the single-agent robotics control and multi-agent highway merging tasks.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 13 of its 40 references that are in the map. First posted on arXiv 19 Feb 2022.

Builds on

13 of 40 references.

References · 13

  1. 2016OpenAI Gymcited by 1,684
  2. 2019Solving Rubik's Cube with a Robot Handcited by 699
  3. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  4. 2017Robust Adversarial Reinforcement Learningcited by 439
  5. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  6. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  7. 2017Learning with Opponent-Learning Awarenesscited by 218
  8. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  9. 2020Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluationcited by 54
  10. 2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
  11. 2020Delay-Aware Model-Based Reinforcement Learning for Continuous Controlcited by 33
  12. 2020Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processescited by 23
1 more
  1. 2019Implicit competitive regularization in GANscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 5; 2024: 9; 2025: 1; 2026: 3.

26 papers.

Sort:

By area:

  1. 2022Robust Reinforcement Learning using Offline Datacited by 95
  2. 2023How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligencecited by 55
  3. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  4. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  5. 2022Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptationcited by 28
  6. 2022Performative Reinforcement Learningcited by 25
  7. 2024CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectoriescited by 11
  8. 2022Continual Vision-based Reinforcement Learning with Group Symmetriescited by 10
  9. 2024Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identificationscited by 10
  10. 2024Safe Multi-Agent Reinforcement Learning with Bilevel Optimization in Autonomous Drivingcited by 9
  11. 2023Improve Robustness of Reinforcement Learning against Observation Perturbations via l_∞ Lipschitz Policy Networkscited by 7
  12. 2022Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variablescited by 6
14 more
  1. 2024MAGICS: Adversarial RL with Minimax Actors Guided by Implicit Critic Stackelberg for Convergent Neural Synthesis of Robot Safetycited by 5
  2. 2026Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environmentscited by 4
  3. 2022A Survey of Decision Making in Adversarial Gamescited by 3
  4. 2023Stackelberg Driver Model for Continual Policy Improvement in Scenario-Based Closed-Loop Autonomous Drivingcited by 3
  5. 2026Distributed General-Purpose Agent Networks: Architecture, Key Mechanisms, and Prototypescited by 3
  6. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learningcited by 2
  7. 2024Rethinking Teacher-Student Curriculum Learning through the Cooperative Mechanics of Experiencecited by 1
  8. 2024VCAT: Vulnerability-aware and Curiosity-driven Adversarial Training for Enhancing Autonomous Vehicle Robustnesscited by 1
  9. 2026Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Surveycited by 1
  10. 2023Zero-shot Transfer Learning of Driving Policy via Socially Adversarial Traffic Flowcited by 0
  11. 2023Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbationscited by 0
  12. 2024Leveraging Noisy Observations in Zero-Sum Gamescited by 0
  13. 2024ReLExS: Reinforcement Learning Explanations for Stackelberg No-Regret Learnerscited by 0
  14. 2024Constrained Optimal Fuel Consumption of HEVs under Observational Noisecited by 0