mlsanityData to 7 Oct 2026

An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning

arXiv 2409.03052 ↗First posted 4 Sep 2024cs.LG cs.MAcited by 46

Not on the map. Why?

Abstract (arXiv)

Multi-agent reinforcement learning (MARL) has exploded in popularity in recent years. Many approaches have been developed but they can be divided into three main types: centralized training and execution (CTE), centralized training for decentralized execution (CTDE), and Decentralized training and execution (DTE). CTDE methods are the most common as they can use centralized information during training but execute in a decentralized manner -- using only information available to that agent during execution. CTDE is the only paradigm that requires a separate training phase where any available information (e.g., other agent policies, underlying states) can be used. As a result, they can be more scalable than CTE methods, do not require communication during execution, and can often perform well. CTDE fits most naturally with the cooperative case, but can be potentially applied in competitive or mixed settings depending on what information is assumed to be observed. This text is an introduction to CTDE in cooperative MARL. It is meant to explain the setting, basic concepts, and common methods. It does not cover all work in CTDE MARL as the subarea is quite extensive. I have included work that I believe is important for understanding the main concepts in the subarea and apologize to those that I have omitted.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 43 of its 73 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

43 of 73 references.

References · 43

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  6. 2016HyperNetworkscited by 1,003
  7. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  8. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  9. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  10. 2019The StarCraft Multi-Agent Challengecited by 589
  11. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  12. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
31 more
  1. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  2. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  3. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  4. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  5. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  6. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 225
  7. 2017Learning with Opponent-Learning Awarenesscited by 218
  8. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  9. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 180
  10. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  11. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  12. 2017Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observabilitycited by 143
  13. 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 114
  14. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  15. 2019Learning Nearly Decomposable Value Functions Via Communication Minimizationcited by 85
  16. 2020Qatten: A General Framework for Cooperative Multiagent Reinforcement Learningcited by 81
  17. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  18. 2021Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learningcited by 54
  19. 2019Is the Policy Gradient a Gradient?cited by 40
  20. 2022COLA: Consistent Learning with Opponent-Learning Awarenesscited by 38
  21. 2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 35
  22. 2013On the Complexity of Solving Markov Decision Problemscited by 31
  23. 2021Unbiased Asymmetric Reinforcement Learning under Partial Observabilitycited by 24
  24. 2023From Explicit Communication to Tacit Cooperation:A Novel Paradigm for Cooperative MARLcited by 11
  25. 2022A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learningcited by 9
  26. 2024On Centralized Critics in Multi-Agent Reinforcement Learningcited by 9
  27. 2019Decentralized Multi-Agents by Imitation of a Centralized Controllercited by 5
  28. 2022MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learningcited by 5
  29. 2024An Initial Introduction to Cooperative Multi-Agent Reinforcement Learningcited by 5
  30. 2024On Stateful Value Factorization in Multi-Agent Reinforcement Learningcited by 5
  31. 2023Best Possible Q-Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 27.

46 papers.

Sort:

By area:

  1. 2025LLM Collaboration With Multi-Agent Reinforcement Learningcited by 47
  2. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  3. 2025A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectivescited by 27
  4. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  5. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 7
  6. 2026The Value of Variance: Mitigating Debate Collapse in Multi-Agent Systems via Uncertainty-Driven Policy Optimizationcited by 6
  7. 2025Low-Rank Agent-Specific Adaptation (LoRASA) for Multi-Agent Policy Learningcited by 4
  8. 2026CHORUS: Decentralized Multi-Embodiment Collaboration with One VLA Policycited by 4
  9. 2025URB -- Urban Routing Benchmark for RL-equipped Connected Autonomous Vehiclescited by 3
  10. 2026Best Practice Critic Optimizationcited by 3
  11. 2026Small Agent Group is the Future of Digital Healthcited by 2
  12. 2026Socially-Weighted Alignment: A Game-Theoretic Framework for Multi-Agent LLM Systemscited by 2
34 more
  1. 2026Action-Graph Policies: Learning Action Co-dependencies in Multi-Agent Reinforcement Learningcited by 2
  2. 2025Beyond Shallow Behavior: Task-Efficient Value-Based Multi-Task Offline MARL via Skill Discoverycited by 1
  3. 2025Overcoming Environmental Meta-Stationarity in MARL via Adaptive Curriculum and Counterfactual Group Advantagecited by 1
  4. 2025Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learningcited by 1
  5. 2025From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demoscited by 1
  6. 2026Communication Methods in Multi-Agent Reinforcement Learningcited by 1
  7. 2026Plasticity-Enhanced Multi-Agent Mixture of Experts for Dynamic Objective Adaptation in UAVs-Assisted Emergency Communication Networkscited by 1
  8. 2026Self-CTRL: Self-Consistency Training with Reinforcement Learningcited by 1
  9. 2024Efficient Training in Multi-Agent Reinforcement Learning: A Communication-Free Framework for the Box-Pushing Problemcited by 0
  10. 2025Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learningcited by 0
  11. 2025A Multi-Agent Reinforcement Learning Approach for Cooperative Air-Ground-Human Crowdsensing in Emergency Rescuecited by 0
  12. 2025LOD: Latent Objective Discovery in Heterogeneous Multi-Agent Reinforcement Learningcited by 0
  13. 2025ReCoDe: Reinforcement Learning-based Dynamic Constraint Design for Multi-Agent Coordinationcited by 0
  14. 2025Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learningcited by 0
  15. 2025Distributed Area Coverage with High Altitude Balloons Using Multi-Agent Reinforcement Learningcited by 0
  16. 2025Quantum-Inspired Multi Agent Reinforcement Learning for Exploration Exploitation Optimization in UAV-Assisted 6G Network Deploymentcited by 0
  17. 2025Reinforcement Networks: novel framework for collaborative Multi-Agent Reinforcement Learning taskscited by 0
  18. 2026Co-jump: Cooperative Jumping with Quadrupedal Robots via Multi-Agent Reinforcement Learningcited by 0
  19. 2026Agentic-Kube: A Graph-Enhanced Multi-Agent Reinforcement Learning Framework for Multi-Objective Kubernetes Schedulingcited by 0
  20. 2026MA-VLCM: A Vision Language Critic Model for Value Estimation of Policies in Multi-Agent Team Settingscited by 0
  21. 2026In Trust We Survive: Emergent Trust Learningcited by 0
  22. 2026Do LLM-derived graph priors improve multi-agent coordination?cited by 0
  23. 2026Cross-Modal Navigation with Multi-Agent Reinforcement Learningcited by 0
  24. 2026PC3D: Zero-Shot Cooperation Across Variable Rosters via Personalized Context Distillationcited by 0
  25. 2026Probabilistic Verification of Recurrent Neural Networks for Single and Multi-Agent Reinforcement Learningcited by 0
  26. 2026Beyond Partner Diversity: An Influence-Based Team Steering Framework for Zero-Shot Human-Machine Teamingcited by 0
  27. 2026SwarmHarness: Skill-Based Task Routing via Decentralized Incentive-Aligned AI Agent Networkscited by 0
  28. 2026Merging model-based control with multi-agent reinforcement learning for multi-agent cooperative teaming strategiescited by 0
  29. 2026PRIME: Plasticity Recovery in Multi-Agent Environments for UAV-Assisted Emergency Communication Networkscited by 0
  30. 2026Reinformed Dreamer: An Asymmetric World Model Efficiently Trained through Latent Guidancecited by 0
  31. 2026FedCritic-MIMO: Communication-Efficient Serverless Federated Critic Learning for Massive-MIMO Resource Control in Open and Disaggregated 6G RANscited by 0
  32. 2026Distributed Optimization of Modular Production Systems using Model-based Reinforcement Learning with Inverse Modelscited by 0
  33. 2026A Decentralized Partially Observable Team Decision Methodology with Delayed Information Sharingcited by 0
  34. 2026Improving LLM Collaboration via Multi-Agent Preference Learningcited by 0