mlsanityData to 14 Oct 2026

K-Level Policy Gradients for Multi-Agent Reinforcement Learning

arXiv 2509.12117 ↗First posted 15 Sep 2025cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Actor-critic algorithms for deep multi-agent reinforcement learning (MARL) typically employ a policy update that responds to the current strategies of other agents. While being straightforward, this approach does not account for the updates of other agents at the same update step, resulting in miscoordination. In this paper, we introduce the K-Level Policy Gradient (KPG), a method that recursively updates each agent against the updated policies of other agents, speeding up the discovery of effective coordinated policies. We theoretically prove that KPG with finite iterates achieves monotonic convergence to a local Nash equilibrium under certain conditions. We provide principled implementations of KPG by applying it to the deep MARL algorithms MAPPO, MADDPG, and FACMAC. Empirically, we demonstrate superior performance over existing deep MARL algorithms in StarCraft II and multi-agent MuJoCo.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 15 Sep 2025.

Builds on

31 of 52 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Generative Adversarial Networkscited by 5,627
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  5. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  6. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  7. 2019The StarCraft Multi-Agent Challengecited by 597
  8. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  9. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  10. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  11. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  12. 2017Learning with Opponent-Learning Awarenesscited by 222
19 more
  1. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 197
  2. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  3. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  4. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  5. 2020Deep Reinforcement Learning for Intelligent Transportation Systems: A Surveycited by 98
  6. 2016Opponent Modeling in Deep Reinforcement Learningcited by 97
  7. 2021Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learningcited by 56
  8. 2022COLA: Consistent Learning with Opponent-Learning Awarenesscited by 38
  9. 2022K-level Reasoning for Zero-Shot Coordination in Hanabicited by 32
  10. 2020Parameter-Based Value Functionscited by 24
  11. 2022Proximal Learning With Opponent-Learning Awarenesscited by 20
  12. 2023Scaling Opponent Shaping to High Dimensional Gamescited by 12
  13. 2020Reinforcement Learning with Iterative Reasoning for Merging in Dense Trafficcited by 10
  14. 2020Driver Modeling through Deep Reinforcement Learning and Behavioral Game Theorycited by 8
  15. 2023Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learningcited by 4
  16. 2021Modeling the effects of environmental and perceptual uncertainty using deterministic reinforcement learning dynamics with partial observabilitycited by 3
  17. 2022Recursive Reasoning in Minimax Games: A Level k Gradient Play Methodcited by 2
  18. 2023Game Theoretic Decision Making by Actively Learning Human Intentions Applied on Autonomous Drivingcited by 2
  19. 2024Applications of Multi-Agent Deep Reinforcement Learning Communication in Network Management: A Surveycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Auton Agentic AI Frameworkcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Natural Policy Gradient and Actor Critic Methods for Constrained Multi-Task Reinforcement Learning
  2. 2025Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective
  3. 2024MARL-LNS: Cooperative Multi-agent Reinforcement Learning via Large Neighborhoods Search
  4. 2025Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning
  5. 2026ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning
  6. 2026Hierarchical Lead Critic based Multi-Agent Reinforcement Learning
  7. 2025Multi-Agent Guided Policy Search for Non-Cooperative Dynamic Games
  8. 2026Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning