mlsanityData to 7 Oct 2026

Coordinated Proximal Policy Optimization

arXiv 2111.04051 ↗First posted 7 Nov 2021cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

We present Coordinated Proximal Policy Optimization (CoPPO), an algorithm that extends the original Proximal Policy Optimization (PPO) to the multi-agent setting. The key idea lies in the coordinated adaptation of step size during the policy update process among multiple agents. We prove the monotonicity of policy improvement when optimizing a theoretically-grounded joint objective, and derive a simplified optimization objective based on a set of approximations. We then interpret that such an objective in CoPPO can achieve dynamic credit assignment among agents, thereby alleviating the high variance issue during the concurrent update of agent policies. Finally, we demonstrate that CoPPO outperforms several strong baselines and is competitive with the latest multi-agent PPO method (i.e. MAPPO) under typical multi-agent settings, including cooperative matrix games and the StarCraft II micromanagement tasks.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 7 Nov 2021.

Builds on

21 of 36 references.

References · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  7. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  8. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  9. 2019The StarCraft Multi-Agent Challengecited by 589
  10. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  11. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  12. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
9 more
  1. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 180
  2. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  3. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  4. 2017A Survey of Learning in Multiagent Environments: Dealing with Non-Stationaritycited by 93
  5. 2019Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learningcited by 90
  6. 2020Towards Playing Full MOBA Games with Deep Reinforcement Learningcited by 78
  7. 2020Value-Decomposition Multi-Agent Actor-Criticscited by 47
  8. 2020Multi-Agent Trust Region Policy Optimizationcited by 19
  9. 2019Convergence of Multi-Agent Learning with a Finite Step Size in General-Sum Gamescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 5; 2024: 3; 2025: 0; 2026: 4.

16 papers.

Sort:

  1. 2023Heterogeneous-Agent Reinforcement Learningcited by 67
  2. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  3. 2023Order Matters: Agent-by-agent Policy Optimizationcited by 24
  4. 2023Maximum Entropy Heterogeneous-Agent Reinforcement Learningcited by 19
  5. 2022Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualizationcited by 13
  6. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Designcited by 11
  7. 2023Multi-agent Attacks for Black-box Social Recommendationscited by 9
  8. 2022Interaction Pattern Disentangling for Multi-Agent Reinforcement Learningcited by 8
  9. 2022Decentralized Policy Optimizationcited by 5
  10. 2023FP3O: Enabling Proximal Policy Optimization in Multi-Agent Cooperation with Parameter-Sharing Versatilitycited by 1
  11. 2024MARL-LNS: Cooperative Multi-agent Reinforcement Learning via Large Neighborhoods Searchcited by 0
  12. 2024Achieving Collective Welfare in Multi-Agent Reinforcement Learning via Suggestion Sharingcited by 0
4 more
  1. 2026Explicit Credit Assignment through Local Rewards and Dependence Graphs in Multi-Agent Reinforcement Learningcited by 0
  2. 2026Robust Instruction Compliance in Cooperative Multi-Agent Reinforcement Learningcited by 0
  3. 2026Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learningcited by 0
  4. 2026OFBD: Object-Focused Background Debiasing for Long-Tailed Learningcited by 0