mlsanityData to 7 Oct 2026

Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning

arXiv 2102.04402 ↗First posted 8 Feb 2021cs.LG cs.AIcited by 54

Not on the map. Why?

Abstract (arXiv)

Centralized Training for Decentralized Execution, where agents are trained offline using centralized information but execute in a decentralized manner online, has gained popularity in the multi-agent reinforcement learning community. In particular, actor-critic methods with a centralized critic and decentralized actors are a common instance of this idea. However, the implications of using a centralized critic in this context are not fully discussed and understood even though it is the standard choice of many algorithms. We therefore formally analyze centralized and decentralized critic approaches, providing a deeper understanding of the implications of critic choice. Because our theory makes unrealistic assumptions, we also empirically compare the centralized and decentralized critic methods over a wide set of environments to validate our theories and to provide practical advice. We show that there exist misconceptions regarding centralized critics in the current literature and show that the centralized critic design is not strictly beneficial, but rather both centralized and decentralized critics have different pros and cons that should be taken into account by algorithm designers.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 24 of its 54 references that are in the map. First posted on arXiv 8 Feb 2021.

Builds on

24 of 54 references.

References · 24

  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  2. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  3. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  4. 2019The StarCraft Multi-Agent Challengecited by 589
  5. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  6. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  7. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  8. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  9. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 257
  10. 2018TarMAC: Targeted Multi-Agent Communicationcited by 201
  11. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  12. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
12 more
  1. 2018Learning Attentional Communication for Multi-Agent Cooperationcited by 170
  2. 2017Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observabilitycited by 143
  3. 2020ROMA: Multi-Agent Reinforcement Learning with Emergent Rolescited by 135
  4. 2019Learning Nearly Decomposable Value Functions Via Communication Minimizationcited by 85
  5. 2019Shapley Q-value: A Local Reward Approach to Solve Global Reward Gamescited by 71
  6. 2018Learning to Teach in Cooperative Multiagent Reinforcement Learningcited by 45
  7. 2018CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learningcited by 31
  8. 2013Nash Convergence of Gradient Dynamics in Iterated General-Sum Gamescited by 18
  9. 2020Macro-Action-Based Deep Multi-Agent Reinforcement Learningcited by 18
  10. 2019Option-Critic in Cooperative Multi-agent Systemscited by 17
  11. 2016Learning to Play Guess Who? and Inventing a Grounded Language as a Consequencecited by 15
  12. 2019Learning Multi-Robot Decentralized Macro-Action-Based Policies via a Centralized Q-Netcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 11; 2023: 13; 2024: 11; 2025: 6; 2026: 5.

54 papers.

Sort:

By area:

  1. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  2. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 59
  3. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 46
  4. 2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
  5. 2021Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysiscited by 25
  6. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problemcited by 18
  7. 2022Asynchronous Actor-Critic for Multi-Agent Reinforcement Learningcited by 16
  8. 2024Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defensecited by 16
  9. 2023Context-aware Communication for Multi-agent Reinforcement Learningcited by 13
  10. 2022Trust Region Bounds for Decentralized PPO Under Non-stationaritycited by 11
  11. 2021On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learningcited by 10
  12. 2023Multi-agent Continual Coordination via Progressive Task Contextualizationcited by 10
38 more
  1. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimationcited by 10
  2. 2024Towards Collaborative Intelligence: Propagating Intentions and Reasoning for Multi-Agent Coordination with Large Language Modelscited by 10
  3. 2021LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammatescited by 9
  4. 2022A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learningcited by 9
  5. 2023Learning Multi-Agent Communication with Contrastive Learningcited by 9
  6. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluationcited by 9
  7. 2024On Centralized Critics in Multi-Agent Reinforcement Learningcited by 9
  8. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learningcited by 7
  9. 2025Exponential Topology-enabled Scalable Communication in Multi-agent Reinforcement Learningcited by 7
  10. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 7
  11. 2023MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignmentcited by 6
  12. 2022Multi-Agent Reinforcement Learning with Shared Resources for Inventory Managementcited by 5
  13. 2024Dynamic Co-Optimization Compiler: Leveraging Multi-Agent Reinforcement Learning for Enhanced DNN Accelerator Performancecited by 4
  14. 2021Local Advantage Actor-Critic for Robust Multi-Agent Deep Reinforcement Learningcited by 3
  15. 2022Bilateral Deep Reinforcement Learning Approach for Better-than-human Car Following Modelcited by 3
  16. 2023Robust Multi-agent Communication via Multi-view Message Certificationcited by 3
  17. 2021Off-Policy Correction For Multi-Agent Reinforcement Learningcited by 2
  18. 2022AdverSAR: Adversarial Search and Rescue via Multi-Agent Reinforcement Learningcited by 2
  19. 2023Efficient Communication via Self-supervised Information Aggregation for Online and Offline Multi-agent Reinforcement Learningcited by 2
  20. 2025Decentralized Reinforcement Learning for Multi-Agent Multi-Resource Allocation via Dynamic Cluster Agreementscited by 2
  21. 2022Hierarchical Reinforcement Learning with Opponent Modeling for Distributed Multi-agent Cooperationcited by 1
  22. 2022CURO: Curriculum Learning for Relative Overgeneralizationcited by 1
  23. 2024A Multi-Agent Reinforcement Learning Testbed for Cognitive Radio Applicationscited by 1
  24. 2025Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learningcited by 1
  25. 2025K-Level Policy Gradients for Multi-Agent Reinforcement Learningcited by 1
  26. 2026Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)cited by 1
  27. 2021Local Advantage Networks for Cooperative Multi-Agent Reinforcement Learningcited by 0
  28. 2022General sum stochastic games with networked information flowscited by 0
  29. 2022Distributed Influence-Augmented Local Simulators for Parallel MARL in Large Networked Systemscited by 0
  30. 2023Traffic Shaping and Hysteresis Mitigation Using Deep Reinforcement Learning in a Connected Driving Environmentcited by 0
  31. 2023Interpretability for Conditional Coordinated Behavior in Multi-Agent Reinforcement Learningcited by 0
  32. 2023An Algorithm For Adversary Aware Decentralized Networked MARLcited by 0
  33. 2023Robust Network Slicing: Multi-Agent Policies, Adversarial Attacks, and Defensive Strategiescited by 0
  34. 2024Centralized vs. Decentralized Multi-Agent Reinforcement Learning for Enhanced Control of Electric Vehicle Charging Networkscited by 0
  35. 2024DCMAC: Demand-aware Customized Multi-Agent Communication via Upper Bound Trainingcited by 0
  36. 2024OPTIMA: Optimized Policy for Intelligent Multi-Agent Systems Enables Coordination-Aware Autonomous Vehiclescited by 0
  37. 2024Signaling and Social Learning in Swarms of Robotscited by 0
  38. 2025Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policiescited by 0

50 most cited of 54 All 54 (JSON)