mlsanityData to 14 Oct 2026

Counterfactual Multi-Agent Policy Gradients

arXiv 1705.08926 ↗First posted 24 May 2017cs.AI cs.MAcited by 751

Not on the map. Why?

Cited by 59 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Cooperative multi-agent systems can be naturally used to model many real world problems, such as network packet routing and the coordination of autonomous vehicles. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent's action, while keeping the other agents' actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actor-critic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controllers that get access to the full state.

Key facts. Cited by 751 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 136; 2022: 137; 2023: 121; 2024: 87; 2025: 116; 2026: 152.

751 papers.

Sort:

By area:

  1. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  2. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
  3. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 155
  4. 2022A Survey of Multi-Agent Deep Reinforcement Learning with Communicationcited by 104
  5. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 98
  6. 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 87
  7. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  8. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  9. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  10. 2024Simplifying Deep Temporal Difference Learningcited by 78
  11. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 75
  12. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 71
38 more
  1. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 70
  2. 2023Heterogeneous-Agent Reinforcement Learningcited by 69
  3. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  4. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 61
  5. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  6. 2021ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mindcited by 58
  7. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  8. 2021Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learningcited by 56
  9. 2021Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networkscited by 54
  10. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 52
  11. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 50
  12. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 49
  13. 2024Learning Multi-Agent Communication from Graph Modeling Perspectivecited by 48
  14. 2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
  15. 2023A Survey on Causal Reinforcement Learningcited by 45
  16. 2025Language Self-Play For Data-Free Trainingcited by 43
  17. 2021Deep Multiagent Reinforcement Learning: Challenges and Directionscited by 42
  18. 2021UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformerscited by 39
  19. 2021Shapley Counterfactual Credits for Multi-Agent Reinforcement Learningcited by 38
  20. 2021Independent Natural Policy Gradient Always Converges in Markov Potential Gamescited by 37
  21. 2021Learning to Ground Multi-Agent Communication with Autoencoderscited by 37
  22. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 37
  23. 2025Multi-Agent Coordination across Diverse Applications: A Surveycited by 37
  24. 2023Heterogeneous Multi-Robot Reinforcement Learningcited by 36
  25. 2023Robust Multi-Agent Reinforcement Learning with State Uncertaintycited by 36
  26. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  27. 2021Multi-Agent Constrained Policy Optimisationcited by 35
  28. 2023Dynamic Handover: Throw and Catch with Bimanual Handscited by 35
  29. 2021XDO: A Double Oracle Algorithm for Extensive-Form Gamescited by 34
  30. 2021Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Surveycited by 34
  31. 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
  32. 2021A New Formalism, Method and Open Issues for Zero-Shot Coordinationcited by 33
  33. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 33
  34. 2021Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimizationcited by 33
  35. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 33
  36. 2021Distributed Heuristic Multi-Agent Path Finding with Communicationcited by 32
  37. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  38. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 32

50 most cited of 751 All 751 (JSON)