mlsanityData to 7 Oct 2026

The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games

arXiv 2103.01955 ↗First posted 2 Mar 2021cs.LG cs.AI cs.MAcited by 903

Not on the map. Why?

Cited by 89 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Proximal Policy Optimization (PPO) is a ubiquitous on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due to the belief that PPO is significantly less sample efficient than off-policy methods in multi-agent systems. In this work, we carefully study the performance of PPO in cooperative multi-agent settings. We show that PPO-based multi-agent algorithms achieve surprisingly strong performance in four popular multi-agent testbeds: the particle-world environments, the StarCraft multi-agent challenge, Google Research Football, and the Hanabi challenge, with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. Importantly, compared to competitive off-policy methods, PPO often achieves competitive or superior results in both final returns and sample efficiency. Finally, through ablation studies, we analyze implementation and hyperparameter factors that are critical to PPO's empirical performance, and give concrete practical suggestions regarding these factors. Our results show that when using these practices, simple PPO-based methods can be a strong baseline in cooperative multi-agent reinforcement learning. Source code is released at urlhttps://github.com/marlbenchmark/on-policy.

Key facts. Cited by 903 papers in the map (data through 7 Oct 2026). Builds on 26 of its 53 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

26 of 53 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  8. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  9. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  10. 2019The StarCraft Multi-Agent Challengecited by 589
  11. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  12. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
14 more
  1. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  2. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  3. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  4. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 257
  5. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  6. 2017Learning with Opponent-Learning Awarenesscited by 218
  7. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 207
  8. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  9. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 194
  10. 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
  11. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  12. 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 85
  13. 2019Simplified Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 47
  14. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 72; 2023: 147; 2024: 166; 2025: 257; 2026: 239.

903 papers.

Sort:

By area:

  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  3. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 148
  4. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  5. 2023Theory of Mind for Multi-Agent Collaboration via Large Language Modelscited by 129
  6. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  7. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  8. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
  9. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
  10. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 73
  11. 2024Simplifying Deep Temporal Difference Learningcited by 71
  12. 2023Heterogeneous-Agent Reinforcement Learningcited by 67
38 more
  1. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  2. 2022VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learningcited by 63
  3. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 60
  4. 2023TorchRL: A data-driven decision-making library for PyTorchcited by 59
  5. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 59
  6. 2021ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mindcited by 57
  7. 2022Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real worldcited by 56
  8. 2022Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergencecited by 54
  9. 2022End-to-End Affordance Learning for Robotic Manipulationcited by 54
  10. 2021Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networkscited by 52
  11. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
  12. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  13. 2021The Power of Exploiter: Provable Multi-Agent RL in Large State Spacescited by 48
  14. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  15. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 47
  16. 2024Learning Multi-Agent Communication from Graph Modeling Perspectivecited by 47
  17. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 46
  18. 2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
  19. 2024CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamicscited by 43
  20. 2024GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPScited by 43
  21. 2025Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learningcited by 39
  22. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  23. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  24. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  25. 2023Robust Multi-Agent Reinforcement Learning with State Uncertaintycited by 35
  26. 2022Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregationcited by 34
  27. 2023Dynamic Handover: Throw and Catch with Bimanual Handscited by 33
  28. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
  29. 2023BenchMARL: Benchmarking Multi-Agent Reinforcement Learningcited by 31
  30. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
  31. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  32. 2021Learning Graphon Mean Field Games and Approximate Nash Equilibriacited by 28
  33. 2023OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Controlcited by 28
  34. 2023Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Frameworkcited by 28
  35. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  36. 2022Machine Learning-Aided Operations and Communications of Unmanned Aerial Vehicles: A Contemporary Surveycited by 27
  37. 2023Learn to Follow: Decentralized Lifelong Multi-agent Pathfinding via Planning and Learningcited by 27
  38. 2023A Review of Cooperation in Multi-agent Learningcited by 27

50 most cited of 903 All 903 (JSON)