mlsanityData to 7 Oct 2026

Settling the Variance of Multi-Agent Policy Gradients

arXiv 2108.08612 ↗First posted 19 Aug 2021cs.LG cs.AI cs.MAcited by 45

Not on the map. Why?

Abstract (arXiv)

Policy gradient (PG) methods are popular reinforcement learning (RL) methods where a baseline is often applied to reduce the variance of gradient estimates. In multi-agent RL (MARL), although the PG theorem can be naturally extended, the effectiveness of multi-agent PG (MAPG) methods degrades as the variance of gradient estimates increases rapidly with the number of agents. In this paper, we offer a rigorous analysis of MAPG methods by, firstly, quantifying the contributions of the number of agents and agents' explorations to the variance of MAPG estimators. Based on this analysis, we derive the optimal baseline (OB) that achieves the minimal variance. In comparison to the OB, we measure the excess variance of existing MARL algorithms such as vanilla MAPG and COMA. Considering using deep neural networks, we also propose a surrogate version of OB, which can be seamlessly plugged into any existing PG methods in MARL. On benchmarks of Multi-Agent MuJoCo and StarCraft challenges, our OB technique effectively stabilises training and improves the performance of multi-agent PPO and COMA algorithms by a significant margin.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 19 Aug 2021.

Builds on

33 of 50 references.

References · 33

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  8. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  9. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  10. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  11. 2019The StarCraft Multi-Agent Challengecited by 589
  12. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
21 more
  1. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  2. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  3. 2018Mean Field Multi-Agent Reinforcement Learningcited by 202
  4. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  5. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  6. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  7. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  8. 2020Game-Theoretic Multiagent Reinforcement Learningcited by 92
  9. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 84
  10. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
  11. 2019Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learningcited by 68
  12. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
  13. 2019Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learningcited by 57
  14. 2021Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learningcited by 54
  15. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 46
  16. 2019Bi-level Actor-Critic for Multi-agent Coordinationcited by 38
  17. 2020Multi-Agent Determinantal Q-Learningcited by 36
  18. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  19. 2021Learning in Nonzero-Sum Stochastic Games with Potentialscited by 25
  20. 2019Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoningcited by 24
  21. 2018Factorized Q-Learning for Large-Scale Multi-Agent Systemscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 10; 2023: 3; 2024: 7; 2025: 14; 2026: 8.

45 papers.

Sort:

By area:

  1. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  2. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  3. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  4. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  5. 2023Heterogeneous-Agent Reinforcement Learningcited by 67
  6. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
  7. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  8. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  9. 2021Multi-Agent Constrained Policy Optimisationcited by 32
  10. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
  11. 2023Maximum Entropy Heterogeneous-Agent Reinforcement Learningcited by 19
  12. 2024Reinforcing Language Agents via Policy Optimization with Action Decompositioncited by 16
33 more
  1. 2022ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependencycited by 15
  2. 2022Heterogeneous-Agent Mirror Learning: A Continuum of Solutions to Cooperative MARLcited by 12
  3. 2024HyperMARL: Adaptive Hypernetworks for Multi-Agent RLcited by 10
  4. 2024Safe Multi-Agent Reinforcement Learning with Bilevel Optimization in Autonomous Drivingcited by 9
  5. 2022Interaction Pattern Disentangling for Multi-Agent Reinforcement Learningcited by 8
  6. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorizationcited by 7
  7. 2022Understanding Value Decomposition Algorithms in Deep Cooperative Multi-Agent Reinforcement Learningcited by 6
  8. 2025Joint Computation Offloading and Resource Allocation for Uncertain Maritime MEC via Cooperation of UAVs and Vesselscited by 3
  9. 2025Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learningcited by 2
  10. 2022Taming Multi-Agent Reinforcement Learning with Estimator Variance Reductioncited by 1
  11. 2024Multi-agent transformer-accelerated RL for satisfaction of STL specificationscited by 1
  12. 2025Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learningcited by 1
  13. 2025Unicorn: A Universal and Collaborative Reinforcement Learning Approach Towards Generalizable Network-Wide Traffic Signal Controlcited by 1
  14. 2025Enhancing Sample Efficiency in Multi-Agent RL with Uncertainty Quantification and Selective Explorationcited by 1
  15. 2025AOAD-MAT: Transformer-based multi-agent deep reinforcement learning model considering agents' order of action decisionscited by 1
  16. 2026COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teamscited by 1
  17. 2024Centralized vs. Decentralized Multi-Agent Reinforcement Learning for Enhanced Control of Electric Vehicle Charging Networkscited by 0
  18. 2024Off-OAB: Off-Policy Policy Gradient Method with Optimal Action-Dependent Baselinecited by 0
  19. 2024Cooperative and Asynchronous Transformer-based Mission Planning for Heterogeneous Teams of Mobile Robotscited by 0
  20. 2025Scalable Safe Multi-Agent Reinforcement Learning for Multi-Agent Systemcited by 0
  21. 2025SrSv: Integrating Sequential Rollouts with Sequential Value Estimation for Multi-agent Reinforcement Learningcited by 0
  22. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARLcited by 0
  23. 2025Resolve Highway Conflict in Multi-Autonomous Vehicle Controls with Local State Attentioncited by 0
  24. 2025Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learningcited by 0
  25. 2025Sequence Pathfinder for Multi-Agent Pickup and Delivery in the Warehousecited by 0
  26. 2025Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspectivecited by 0
  27. 2026Explicit Credit Assignment through Local Rewards and Dependence Graphs in Multi-Agent Reinforcement Learningcited by 0
  28. 2026Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learningcited by 0
  29. 2026Hierarchical Lead Critic based Multi-Agent Reinforcement Learningcited by 0
  30. 2026Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learningcited by 0
  31. 2026Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learningcited by 0
  32. 2026Hierarchical Multi-Agent Reinforcement Learning for Carbon-Aware AI Data Centers in Power Distribution Systemscited by 0
  33. 2026Emergent Charging Coordination in Electric Delivery Fleetscited by 0