mlsanityData to 7 Oct 2026

Order Matters: Agent-by-agent Policy Optimization

arXiv 2302.06205 ↗First posted 13 Feb 2023cs.AI cs.GT cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

While multi-agent trust region algorithms have achieved great success empirically in solving coordination tasks, most of them, however, suffer from a non-stationarity problem since agents update their policies simultaneously. In contrast, a sequential scheme that updates policies agent-by-agent provides another perspective and shows strong performance. However, sample inefficiency and lack of monotonic improvement guarantees for each agent are still the two significant challenges for the sequential scheme. In this paper, we propose the Agent-by-agent Policy Optimization (A2PO) algorithm to improve the sample efficiency and retain the guarantees of monotonic improvement for each agent during training. We justify the tightness of the monotonic improvement bound compared with other trust region algorithms. From the perspective of sequentially updating agents, we further consider the effect of agent updating order and extend the theory of non-stationarity into the sequential update scheme. To evaluate A2PO, we conduct a comprehensive empirical study on four benchmarks: StarCraftII, Multi-agent MuJoCo, Multi-agent Particle Environment, and Google Research Football full game scenarios. A2PO consistently outperforms strong baselines.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

36 of 50 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  6. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  8. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  9. 2019The StarCraft Multi-Agent Challengecited by 589
  10. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  11. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  12. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
24 more
  1. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  2. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 225
  3. 2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 209
  4. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  5. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  6. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  7. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  8. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  9. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  10. 2017Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problemscited by 126
  11. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 96
  12. 2017A Survey of Learning in Multiagent Environments: Dealing with Non-Stationaritycited by 93
  13. 2019Truly Proximal Policy Optimizationcited by 68
  14. 2016Q(λ) with Off-Policy Correctionscited by 48
  15. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
  16. 2021MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learningcited by 23
  17. 2020Multi-Agent Trust Region Policy Optimizationcited by 19
  18. 2016A Primer on Coordinate Descent Algorithmscited by 16
  19. 2021Coordinated Proximal Policy Optimizationcited by 16
  20. 2020WD3: Taming the Estimation Bias in Deep Reinforcement Learningcited by 15
  21. 2021A Game-Theoretic Approach to Multi-Agent Trust Region Optimizationcited by 7
  22. 2021Dealing with Non-Stationarity in MARL via Trust-Region Decompositioncited by 5
  23. 2018Accelerating Greedy Coordinate Descent Methodscited by 4
  24. 2021An Analytical Update Rule for General Policy Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 5; 2025: 10; 2026: 2.

24 papers.

Sort:

By area:

  1. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 73
  2. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
  3. 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 23
  4. 2023Maximum Entropy Heterogeneous-Agent Reinforcement Learningcited by 19
  5. 2024Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Taskcited by 17
  6. 2024RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviorscited by 16
  7. 2023Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordinationcited by 12
  8. 2023Optimistic Multi-Agent Policy Gradientcited by 4
  9. 2024Autonomous Goal Detection and Cessation in Reinforcement Learning: A Case Study on Source Term Estimationcited by 4
  10. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4
  11. 2025SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Controlcited by 4
  12. 2024Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Explorationcited by 3
12 more
  1. 2023Stackelberg Decision Transformer for Asynchronous Action Coordination in Multi-Agent Systemscited by 1
  2. 2023FP3O: Enabling Proximal Policy Optimization in Multi-Agent Cooperation with Parameter-Sharing Versatilitycited by 1
  3. 2025PMAT: Optimizing Action Generation Order in Multi-Agent Reinforcement Learningcited by 1
  4. 2025Iterative Multi-Agent Reinforcement Learning: A Novel Approach Toward Real-World Multi-Echelon Inventory Optimizationcited by 1
  5. 2025AOAD-MAT: Transformer-based multi-agent deep reinforcement learning model considering agents' order of action decisionscited by 1
  6. 2024Computing Ex Ante Equilibrium in Heterogeneous Zero-Sum Team Gamescited by 0
  7. 2025SrSv: Integrating Sequential Rollouts with Sequential Value Estimation for Multi-agent Reinforcement Learningcited by 0
  8. 2025CooT: Learning to Coordinate In-Context with Coordination Transformerscited by 0
  9. 2025Remembering the Markov Property in Cooperative MARLcited by 0
  10. 2025Multi-Agent Trust Region Policy Optimisation: A Joint Constraint Approachcited by 0
  11. 2026CHDP: Cooperative Hybrid Diffusion Policies for Reinforcement Learning in Parameterized Action Spacecited by 0
  12. 2026Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learningcited by 0