mlsanityData to 14 Oct 2026

Multi-Agent Guided Policy Optimization

arXiv 2507.18059 ↗First posted 24 Jul 2025cs.AI cs.MAcited by 3

Not on the map. Why?

Abstract (arXiv)

Due to practical constraints such as partial observability and limited communication, Centralized Training with Decentralized Execution (CTDE) has become the dominant paradigm in cooperative Multi-Agent Reinforcement Learning (MARL). However, existing CTDE methods often underutilize centralized training or lack theoretical guarantees. We propose Multi-Agent Guided Policy Optimization (MAGPO), a novel framework that better leverages centralized training by integrating centralized guidance with decentralized execution. MAGPO uses an autoregressive joint policy for scalable, coordinated exploration and explicitly aligns it with decentralized policies to ensure deployability under partial observability. We provide theoretical guarantees of monotonic policy improvement and empirically evaluate MAGPO on 43 tasks across 6 diverse environments. Results show that MAGPO consistently outperforms strong CTDE baselines and matches or surpasses fully centralized approaches, offering a principled and practical solution for decentralized multi-agent learning. Our code and experimental data can be found in https://github.com/liyheng/MAGPO.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 42 of its 60 references that are in the map. First posted on arXiv 24 Jul 2025.

Builds on

42 of 60 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  8. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  10. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  11. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  12. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
30 more
  1. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  2. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 294
  3. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  4. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 238
  5. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  6. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
  7. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  8. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 155
  9. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  10. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  11. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  12. 2020Shared Experience Actor-Critic for Multi-Agent Reinforcement Learningcited by 116
  13. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 98
  14. 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 87
  15. 2020Qatten: A General Framework for Cooperative Multiagent Reinforcement Learningcited by 81
  16. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 49
  17. 2018Deep Reinforcement Learning for Swarm Systemscited by 45
  18. 2018GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithmscited by 40
  19. 2021UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformerscited by 39
  20. 2020Bridging the Imitation Gap by Adaptive Insubordinationcited by 38
  21. 2020Robust Asymmetric Learning in POMDPscited by 23
  22. 2023Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?cited by 18
  23. 2023TGRL: An Algorithm for Teacher Guided Reinforcement Learningcited by 17
  24. 2021Mava: a research library for distributed multi-agent reinforcement learning in JAXcited by 16
  25. 2022PTDE: Personalized Training with Distilled Execution for Multi-Agent Reinforcement Learningcited by 11
  26. 2024Sable: a Performant, Efficient and Scalable Sequence Model for MARLcited by 10
  27. 2023TransfQMix: Transformers for Leveraging the Graph Structure of Multi-Agent Reinforcement Learning Problemscited by 9
  28. 2022CTDS: Centralized Teacher with Decentralized Student for Multi-Agent Reinforcement Learningcited by 7
  29. 2025Guided Policy Optimization under Partial Observabilitycited by 3
  30. 2021Decentralized Circle Formation Control for Fish-like Robots in the Real-world via Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Training-Free Agentic AI: Probabilistic Control and Coordination in Multi-Agent LLM Systemscited by 1
  2. 2026When Does Hierarchy Help? Benchmarking Agent Coordination in Event-Driven Industrial Schedulingcited by 0
  3. 2026Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning
  2. 2022Decentralized Policy Optimization
  3. 2025Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective
  4. 2022Scalable Multi-Agent Model-Based Reinforcement Learning
  5. 2024Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey
  6. 2026Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learning
  7. 2025HCPO: Hierarchical Conductor-Based Policy Optimization in Multi-Agent Reinforcement Learning
  8. 2024An Initial Introduction to Cooperative Multi-Agent Reinforcement Learning