mlsanityData to 7 Oct 2026

MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning

arXiv 2209.08244 ↗First posted 17 Sep 2022cs.LG cs.MAcited by 5

Not on the map. Why?

Abstract (arXiv)

Decentralized learning has shown great promise for cooperative multi-agent reinforcement learning (MARL). However, non-stationarity remains a significant challenge in fully decentralized learning. In the paper, we tackle the non-stationarity problem in the simplest and fundamental way and propose multi-agent alternate Q-learning (MA2QL), where agents take turns updating their Q-functions by Q-learning. MA2QL is a minimalist approach to fully decentralized cooperative MARL but is theoretically grounded. We prove that when each agent guarantees varepsilon-convergence at each turn, their joint policy converges to a Nash equilibrium. In practice, MA2QL only requires minimal changes to independent Q-learning (IQL). We empirically evaluate MA2QL on a variety of cooperative multi-agent tasks. Results show MA2QL consistently outperforms IQL, which verifies the effectiveness of MA2QL, despite such minimal changes.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 17 Sep 2022.

Builds on

26 of 43 references.

References · 26

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  4. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  5. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  6. 2019The StarCraft Multi-Agent Challengecited by 589
  7. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  8. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  9. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  10. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  11. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  12. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 207
14 more
  1. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  2. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 180
  3. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  4. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  5. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  6. 2021Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networkscited by 52
  7. 2019Learning Transferable Cooperative Behavior in Multi-Agent Teamscited by 40
  8. 2019Bi-level Actor-Critic for Multi-agent Coordinationcited by 38
  9. 2017Lenient Multi-Agent Deep Reinforcement Learningcited by 35
  10. 2019A Cooperative Multi-Agent Reinforcement Learning Framework for Resource Balancing in Complex Logistics Networkcited by 22
  11. 2021Divergence-Regularized Multi-Agent Actor-Criticcited by 15
  12. 2020Multiagent Value Iteration Algorithms in Dynamic Programming and Reinforcement Learningcited by 12
  13. 2020F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learningcited by 10
  14. 2018Alternating Randomized Block Coordinate Descentcited by 8

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 1.

5 papers.

  1. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 46
  2. 2023Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?cited by 18
  3. 2024Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Surveycited by 5
  4. 2023Best Possible Q-Learningcited by 4
  5. 2025Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functionscited by 0