mlsanityData to 7 Oct 2026

Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning

arXiv 2109.11251 ↗First posted 23 Sep 2021cs.AI cs.MAcited by 180

Not on the map. Why?

Abstract (arXiv)

Trust region methods rigorously enabled reinforcement learning (RL) agents to learn monotonically improving policies, leading to superior performance on a variety of tasks. Unfortunately, when it comes to multi-agent reinforcement learning (MARL), the property of monotonic improvement may not simply apply; this is because agents, even in cooperative games, could have conflicting directions of policy updates. As a result, achieving a guaranteed improvement on the joint policy where each agent acts individually remains an open challenge. In this paper, we extend the theory of trust region learning to MARL. Central to our findings are the multi-agent advantage decomposition lemma and the sequential policy update scheme. Based on these, we develop Heterogeneous-Agent Trust Region Policy Optimisation (HATPRO) and Heterogeneous-Agent Proximal Policy Optimisation (HAPPO) algorithms. Unlike many existing MARL algorithms, HATRPO/HAPPO do not need agents to share parameters, nor do they need any restrictive assumptions on decomposibility of the joint value function. Most importantly, we justify in theory the monotonic improvement property of HATRPO/HAPPO. We evaluate the proposed methods on a series of Multi-Agent MuJoCo and StarCraftII tasks. Results show that HATRPO and HAPPO significantly outperform strong baselines such as IPPO, MAPPO and MADDPG on all tested tasks, therefore establishing a new state of the art.

Key facts. Cited by 180 papers in the map (data through 7 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 23 Sep 2021.

Builds on

25 of 39 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  7. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  8. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  9. 2019The StarCraft Multi-Agent Challengecited by 589
  10. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  11. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  12. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
13 more
  1. 2020Game-Theoretic Multiagent Reinforcement Learningcited by 92
  2. 2019Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learningcited by 68
  3. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 64
  4. 2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
  5. 2020EigenGame: PCA as a Nash Equilibriumcited by 39
  6. 2019Bi-level Actor-Critic for Multi-agent Coordinationcited by 38
  7. 2020Multi-Agent Determinantal Q-Learningcited by 36
  8. 2021Learning in Nonzero-Sum Stochastic Games with Potentialscited by 25
  9. 2019Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoningcited by 24
  10. 2021MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learningcited by 23
  11. 2020Multi-Agent Trust Region Policy Optimizationcited by 19
  12. 2019Multiagent Rollout Algorithms and Reinforcement Learningcited by 10
  13. 2021A Game-Theoretic Approach to Multi-Agent Trust Region Optimizationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 25; 2023: 33; 2024: 41; 2025: 41; 2026: 36.

180 papers.

Sort:

By area:

  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  2. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  3. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  4. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  5. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 59
  6. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
  7. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  8. 2024Learning Multi-Agent Communication from Graph Modeling Perspectivecited by 47
  9. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 46
  10. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  11. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  12. 2023Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approachcited by 38
38 more
  1. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  2. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  3. 2021Independent Natural Policy Gradient Always Converges in Markov Potential Gamescited by 34
  4. 2021Multi-Agent Constrained Policy Optimisationcited by 32
  5. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
  6. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  7. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  8. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  9. 2023OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Controlcited by 28
  10. 2023A Review of Cooperation in Multi-agent Learningcited by 27
  11. 2025A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectivescited by 27
  12. 2024A Survey on Self-play Methods in Reinforcement Learningcited by 26
  13. 2023Order Matters: Agent-by-agent Policy Optimizationcited by 24
  14. 2022MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Librarycited by 21
  15. 2022More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorizationcited by 19
  16. 2023Maximum Entropy Heterogeneous-Agent Reinforcement Learningcited by 19
  17. 2023Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?cited by 18
  18. 2023Attacking Cooperative Multi-Agent Reinforcement Learning by Adversarial Minority Influencecited by 16
  19. 2024Reinforcing Language Agents via Policy Optimization with Action Decompositioncited by 16
  20. 2022ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependencycited by 15
  21. 2025Generative Models in Decision Making: A Surveycited by 14
  22. 2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 13
  23. 2022Heterogeneous-Agent Mirror Learning: A Continuum of Solutions to Cooperative MARLcited by 12
  24. 2022Trust Region Bounds for Decentralized PPO Under Non-stationaritycited by 11
  25. 2023From Explicit Communication to Tacit Cooperation:A Novel Paradigm for Cooperative MARLcited by 11
  26. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  27. 2022Stateful active facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learningcited by 10
  28. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimationcited by 10
  29. 2024Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Modelscited by 10
  30. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 10
  31. 2024Sable: a Performant, Efficient and Scalable Sequence Model for MARLcited by 10
  32. 2023Cooperative Multi-Agent Reinforcement Learning: Asynchronous Communication and Linear Function Approximationcited by 9
  33. 2023IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARLcited by 9
  34. 2024Multi-Sender Persuasion: A Computational Perspectivecited by 9
  35. 2024Safe Multi-Agent Reinforcement Learning with Bilevel Optimization in Autonomous Drivingcited by 9
  36. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Explorationcited by 9
  37. 2025Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspectivecited by 9
  38. 2022Interaction Pattern Disentangling for Multi-Agent Reinforcement Learningcited by 8

50 most cited of 180 All 180 (JSON)