mlsanityData to 7 Oct 2026

Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms

arXiv 2310.10810 ↗First posted 16 Oct 2023cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Multi-Agent Reinforcement Learning (MARL) has shown promising results across several domains. Despite this promise, MARL policies often lack robustness and are therefore sensitive to small changes in their environment. This presents a serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment. In this work we show that we can gain robustness by controlling a policy's Lipschitz constant, and under mild conditions, establish the existence of a Lipschitz and close-to-optimal policy. Based on these insights, we propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization. The ERNIE framework provides robustness against noisy observations, changing transition dynamics, and malicious actions of agents. However, ERNIE's adversarial regularization may introduce some training instability. To reduce this instability, we reformulate adversarial regularization as a Stackelberg game. We demonstrate the effectiveness of the proposed framework with extensive experiments in traffic light control and particle environments. In addition, we extend ERNIE to mean-field MARL with a formulation based on distributionally robust optimization that outperforms its non-robust counterpart and is of independent interest. Our code is available at https://github.com/abukharin3/ERNIE.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

34 of 57 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016OpenAI Gymcited by 1,684
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  5. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  6. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  7. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  8. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  9. 2017Robust Adversarial Reinforcement Learningcited by 439
  10. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  11. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  12. 2018Mean Field Multi-Agent Reinforcement Learningcited by 202
22 more
  1. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
  2. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  3. 2022Robust Reinforcement Learning using Offline Datacited by 95
  4. 2020Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approachcited by 83
  5. 2021Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Controlcited by 68
  6. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  7. 2019Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learningcited by 57
  8. 2019Risk Averse Robust Adversarial Reinforcement Learningcited by 54
  9. 2019Wasserstein Robust Reinforcement Learningcited by 50
  10. 2020On the Robustness of Cooperative Multi-Agent Reinforcement Learningcited by 39
  11. 2021Policy Smoothing for Provably Robust Reinforcement Learningcited by 39
  12. 2023Robust Multi-Agent Reinforcement Learning with State Uncertaintycited by 35
  13. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
  14. 2021Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approachcited by 27
  15. 2022What is the Solution for State-Adversarial Multi-Agent Reinforcement Learning?cited by 21
  16. 2020Breaking the Curse of Many Agents: Provable Mean Embedding Q-Iteration for Mean-Field Reinforcement Learningcited by 19
  17. 2022RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on statescited by 16
  18. 2021Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approachcited by 13
  19. 2019Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimizationcited by 12
  20. 2021Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approachcited by 7
  21. 2021Self-Training with Differentiable Teachercited by 7
  22. 2021Adversarially Regularized Policy Learning Guided by Trajectory Optimizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 9; 2026: 10.

25 papers.

Sort:

By area:

  1. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  2. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
  3. 2025Seven Security Challenges in Cross-domain Multi-agent LLM Systemscited by 12
  4. 2025Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalizationcited by 11
  5. 2025Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learningcited by 6
  6. 2024The Bandit Whisperer: Communication Learning for Restless Banditscited by 5
  7. 2024Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-errorcited by 4
  8. 2024Robust Deep Reinforcement Learning against Adversarial Behavior Manipulationcited by 2
  9. 2025Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mappingcited by 2
  10. 2025cMALC-D: Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blendingcited by 2
  11. 2026Training Generalizable Collaborative Agents via Strategic Risk Aversioncited by 2
  12. 2025Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learningcited by 1
13 more
  1. 2026Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learningcited by 1
  2. 2024Robustifying a Policy in Multi-Agent RL with Diverse Cooperative Behaviors and Adversarial Style Sampling for Assistive Taskscited by 0
  3. 2025Distributed Detection of Adversarial Attacks in Multi-Agent Reinforcement Learning with Continuous Action Spacecited by 0
  4. 2025DeepForgeSeal: Latent Space-Driven Semi-Fragile Watermarking for Deepfake Detection Using Adversarial Reinforcement Learningcited by 0
  5. 2025On the Tension Between Optimality and Adversarial Robustness in Policy Optimizationcited by 0
  6. 2026Rationality Measurement and Theory for Reinforcement Learning Agentscited by 0
  7. 2026Distributionally Robust Cooperative Multi-Agent Reinforcement Learning via Robust Value Factorizationcited by 0
  8. 2026Shuffling the Data, Stretching the Step-size: Sharper Bias in constant step-size SGDcited by 0
  9. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0
  10. 2026Skillsets on the Chain: A Blockchain-based Zero-Trust Framework for Agentic AI Networkingcited by 0
  11. 2026Online Security Learning in Cooperative Multi-Agent Systems under Hidden Byzantine Attackscited by 0
  12. 2026MARBO: Relational Belief Grounding for LLM Agents in Social Deduction Gamescited by 0
  13. 2026Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentationcited by 0