mlsanityData to 7 Oct 2026

Robust Multi-Agent Reinforcement Learning with State Uncertainty

arXiv 2307.16212 ↗First posted 30 Jul 2023cs.LG cs.AI cs.GTcited by 35

Not on the map. Why?

Abstract (arXiv)

In real-world multi-agent reinforcement learning (MARL) applications, agents may not have perfect state information (e.g., due to inaccurate measurement or malicious attacks), which challenges the robustness of agents' policies. Though robustness is getting important in MARL deployment, little prior work has studied state uncertainties in MARL, neither in problem formulation nor algorithm design. Motivated by this robustness issue and the lack of corresponding studies, we study the problem of MARL with state uncertainty in this work. We provide the first attempt to the theoretical and empirical analysis of this challenging problem. We first model the problem as a Markov Game with state perturbation adversaries (MG-SPA) by introducing a set of state perturbation adversaries into a Markov Game. We then introduce robust equilibrium (RE) as the solution concept of an MG-SPA. We conduct a fundamental analysis regarding MG-SPA such as giving conditions under which such a robust equilibrium exists. Then we propose a robust multi-agent Q-learning (RMAQ) algorithm to find such an equilibrium, with convergence guarantees. To handle high-dimensional state-action space, we design a robust multi-agent actor-critic (RMAAC) algorithm based on an analytical expression of the policy gradient derived in the paper. Our experiments show that the proposed RMAQ algorithm converges to the optimal value function; our RMAAC algorithm outperforms several MARL and robust MARL methods in multiple multi-agent environments when state uncertainty is present. The source code is public on urlhttps://github.com/sihongho/robustmarlwithstateuncertainty.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 30 of its 86 references that are in the map. First posted on arXiv 30 Jul 2023.

Builds on

30 of 86 references.

References · 30

  1. 2021Convex optimizationcited by 2,479
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  6. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  7. 2017Generative Adversarial Networks: An Overviewcited by 588
  8. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  9. 2018Is Q-learning Provably Efficient?cited by 485
  10. 2017A Tutorial on Thompson Samplingcited by 420
  11. 2017Adversarial Attacks on Neural Network Policiescited by 314
  12. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
18 more
  1. 2018Mean Field Multi-Agent Reinforcement Learningcited by 202
  2. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  3. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 163
  4. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
  5. 2017Delving into adversarial attacks on deep policiescited by 104
  6. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  7. 2020Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learningcited by 93
  8. 2020Game-Theoretic Multiagent Reinforcement Learningcited by 92
  9. 2017Boltzmann Exploration Done Rightcited by 69
  10. 2020On the Robustness of Cooperative Multi-Agent Reinforcement Learningcited by 39
  11. 2020Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learningcited by 23
  12. 2021ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxationcited by 17
  13. 2019Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Gamescited by 11
  14. 2020FormulaZero: Distributionally Robust Online Adaptation via Offline Population Synthesiscited by 10
  15. 2022Data-Driven Distributionally Robust Electric Vehicle Balancing for Mobility-on-Demand Systems under Demand and Supply Uncertaintiescited by 8
  16. 2020Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communicationcited by 6
  17. 2022Data-Driven Distributionally Robust Electric Vehicle Balancing for Autonomous Mobility-on-Demand Systems under Demand and Supply Uncertaintiescited by 6
  18. 2020Online Connectivity-aware Dynamic Deployment for Heterogeneous Multi-Robot Systemscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 8; 2025: 14; 2026: 4.

35 papers.

Sort:

By area:

  1. 2023Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithmscited by 25
  2. 2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learningcited by 12
  3. 2023Solving a Class of Non-Convex Minimax Optimization in Federated Learningcited by 10
  4. 2023Robust Multi-Agent Reinforcement Learning by Mutual Information Regularizationcited by 6
  5. 2025Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learningcited by 6
  6. 2026The Value of Variance: Mitigating Debate Collapse in Multi-Agent Systems via Uncertainty-Driven Policy Optimizationcited by 6
  7. 2025Uncertainty in Action: Confidence Elicitation in Embodied Agentscited by 5
  8. 2023R^3: On-device Real-Time Deep Reinforcement Learning for Autonomous Roboticscited by 4
  9. 2023RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamicscited by 4
  10. 2023Robust Communicative Multi-Agent Reinforcement Learning with Active Defensecited by 4
  11. 2024Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-errorcited by 4
  12. 2024Towards Robust Vision Transformer via Masked Adaptive Ensemblecited by 4
23 more
  1. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  2. 2023Serverless Federated AUPRC Optimization for Multi-Party Collaborative Imbalanced Data Miningcited by 3
  3. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversioncited by 3
  4. 2024Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbationscited by 2
  5. 2025cMALC-D: Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blendingcited by 2
  6. 2023Robustness to Multi-Modal Environment Uncertainty in MARL using Curriculum Learningcited by 1
  7. 2024Towards Fault Tolerance in Multi-Agent Reinforcement Learningcited by 1
  8. 2025Microservice Deployment in Space Computing Power Networks via Robust Reinforcement Learningcited by 1
  9. 2025Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learningcited by 1
  10. 2026SphUnc: Hyperspherical Uncertainty Decomposition and Causal Identification via Information Geometrycited by 1
  11. 2026Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learningcited by 1
  12. 2023Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learningcited by 0
  13. 2024α-OCC: Uncertainty-Aware Camera-based 3D Semantic Occupancy Predictioncited by 0
  14. 2024Multi-modal Spatial Clustering for Spatial Transcriptomics Utilizing High-resolution Histology Imagescited by 0
  15. 2024Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Gamescited by 0
  16. 2025Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Delayed Observationcited by 0
  17. 2025Knowledge capture, adaptation and composition (KCAC): A framework for cross-task curriculum learning in robotic manipulationcited by 0
  18. 2025Robust and Safe Multi-Agent Reinforcement Learning with Communication for Autonomous Vehicles: From Simulation to Hardwarecited by 0
  19. 2025Enhancing Privacy in Decentralized Min-Max Optimization: A Differentially Private Approachcited by 0
  20. 2025Diffusion Guided Adversarial State Perturbations in Reinforcement Learningcited by 0
  21. 2025Fault-Tolerant MARL for CAVs under Observation Perturbations for Highway On-Ramp Mergingcited by 0
  22. 2025On the Tension Between Optimality and Adversarial Robustness in Policy Optimizationcited by 0
  23. 2026Distributionally Robust Cooperative Multi-Agent Reinforcement Learning via Robust Value Factorizationcited by 0