mlsanityData to 7 Oct 2026

Communication-Efficient Actor-Critic Methods for Homogeneous Markov Games

arXiv 2202.09422 ↗First posted 18 Feb 2022cs.MA cs.AI cs.GTcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent success in cooperative multi-agent reinforcement learning (MARL) relies on centralized training and policy sharing. Centralized training eliminates the issue of non-stationarity MARL yet induces large communication costs, and policy sharing is empirically crucial to efficient learning in certain tasks yet lacks theoretical justification. In this paper, we formally characterize a subclass of cooperative Markov games where agents exhibit a certain form of homogeneity such that policy sharing provably incurs no suboptimality. This enables us to develop the first consensus-based decentralized actor-critic method where the consensus update is applied to both the actors and the critics while ensuring convergence. We also develop practical algorithms based on our decentralized actor-critic method to reduce the communication cost during training, while still yielding policies comparable with centralized training.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 18 of its 28 references that are in the map. First posted on arXiv 18 Feb 2022.

Builds on

18 of 28 references.

References · 18

  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  3. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  4. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  5. 2016Learning Multiagent Communication with Backpropagationcited by 355
  6. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  7. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  8. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  9. 2019Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Controlcited by 159
  10. 2018Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Taskscited by 114
  11. 2018LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learningcited by 95
  12. 2017Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Gamescited by 93
6 more
  1. 2019Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Controlcited by 46
  2. 2019PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learningcited by 46
  3. 2021Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysiscited by 25
  4. 2020Networked Multi-Agent Reinforcement Learning with Emergent Communicationcited by 12
  5. 2019Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensuscited by 11
  6. 2018Policy Gradient With Value Function Approximation For Collective Multiagent Planningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 2; 2026: 2.

5 papers.

  1. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
  2. 2023rm E(3)-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learningcited by 4
  3. 2025Symmetries-enhanced Multi-Agent Reinforcement Learningcited by 1
  4. 2026Decentralized Multi-agent Reinforcement Learning for Resilient Critical Infrastructurescited by 0
  5. 2026MASkillBlender: Decentralized Whole-Body Coordination for Multi-Humanoid Loco-Manipulation via Skill Blendingcited by 0