mlsanityData to 14 Oct 2026

ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning

arXiv 2606.30072 ↗First posted 29 Jun 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Cooperative tasks in Multi-Agent Reinforcement Learning (MARL) require agents to collectively maximize a shared return. Under the Centralized Training with Decentralized Execution (CTDE) paradigm, policy gradients have remained difficult to compute directly. Prior methods largely follow two approaches: independent factorized updates with centralized critics, which lack general joint-improvement guarantees without value decomposition assumptions, or alternating best-response updates, which can converge to suboptimal Nash Equilibria. In this paper, we show the joint policy gradient admits an exact decentralized decomposition of per-agent terms, each formed from per-agent score functions and decentralized critics. Based on this decomposition, we develop Agent-Chained Policy Optimization (ACPO), where actors are trained independently, with their updates together constituting a single step on the joint policy gradient. Central to this result is a serialized view of the simultaneous joint decision in which agents commit actions one at a time, each conditioning on a belief over preceding actions that ties the independent per-agent updates into a single joint step. We evaluate on-policy and off-policy instantiations of ACPO on Multi-Robot Warehouse, SMACv2, and MA-MuJoCo, where it outperforms strong baselines, with the gap widening as the number of agents grows.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

35 of 48 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  7. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  8. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  9. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  10. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  11. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  12. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
23 more
  1. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  2. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  3. 2019Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Controlcited by 164
  4. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 155
  5. 2017Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problemscited by 131
  6. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 122
  7. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 98
  8. 2023Heterogeneous-Agent Reinforcement Learningcited by 69
  9. 2018Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 68
  10. 2020Agent Modelling under Partial Observability for Deep Reinforcement Learningcited by 57
  11. 2021Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networkscited by 54
  12. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 32
  13. 2021SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learningcited by 26
  14. 2021Multi-Agent MDP Homomorphic Networkscited by 25
  15. 2022MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Librarycited by 21
  16. 2022More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorizationcited by 19
  17. 2023Maximum Entropy Heterogeneous-Agent Reinforcement Learningcited by 19
  18. 2020Multiagent Value Iteration Algorithms in Dynamic Programming and Reinforcement Learningcited by 12
  19. 2024On Centralized Critics in Multi-Agent Reinforcement Learningcited by 10
  20. 2021Cooperative Multi-Agent Fairness and Equivariant Policiescited by 8
  21. 2022CTDS: Centralized Teacher with Decentralized Student for Multi-Agent Reinforcement Learningcited by 7
  22. 2022Towards Global Optimality in Cooperative MARL with the Transformation And Distillation Frameworkcited by 5
  23. 2024Optimally Solving Simultaneous-Move Dec-POMDPs: The Sequential Central Planning Approachcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Agent Guided Policy Optimization
  2. 2025Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective
  3. 2022Decentralized Policy Optimization
  4. 2024Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey
  5. 2026Aggregate in the Advantage, Not the Ratio: A Canonical-Form Analysis of Cooperative Multi-Agent Policy Optimization
  6. 2025Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies
  7. 2026Action-Graph Policies: Learning Action Co-dependencies in Multi-Agent Reinforcement Learning
  8. 2026Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learning