mlsanityData to 7 Oct 2026

Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning

arXiv 2102.03479 ↗First posted 6 Feb 2021cs.LG cs.AI cs.MAcited by 68

Not on the map. Why?

Abstract (arXiv)

Many complex multi-agent systems such as robot swarms control and autonomous vehicle coordination can be modeled as Multi-Agent Reinforcement Learning (MARL) tasks. QMIX, a widely popular MARL algorithm, has been used as a baseline for the benchmark environments, e.g., Starcraft Multi-Agent Challenge (SMAC), Difficulty-Enhanced Predator-Prey (DEPP). Recent variants of QMIX target relaxing the monotonicity constraint of QMIX, allowing for performance improvement in SMAC. In this paper, we investigate the code-level optimizations of these variants and the monotonicity constraint. (1) We find that such improvements of the variants are significantly affected by various code-level optimizations. (2) The experiment results show that QMIX with normalized optimizations outperforms other works in SMAC; (3) beyond the common wisdom from these works, the monotonicity constraint can improve sample efficiency in SMAC and DEPP. We also discuss why monotonicity constraints work well in purely cooperative tasks with a theoretical analysis. We open-source the code at urlhttps://github.com/hijkzzz/pymarl2.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 6 Feb 2021.

Builds on

29 of 40 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  6. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  7. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  8. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  9. 2019The StarCraft Multi-Agent Challengecited by 589
  10. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  11. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  12. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
17 more
  1. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  2. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  3. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  4. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  5. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  6. 2020Phasic Policy Gradientcited by 121
  7. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  8. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  9. 2019Deep Coordination Graphscited by 96
  10. 2020Qatten: A General Framework for Cooperative Multiagent Reinforcement Learningcited by 81
  11. 2020Value-Decomposition Multi-Agent Actor-Criticscited by 47
  12. 2017Guided Deep Reinforcement Learning for Swarm Systemscited by 44
  13. 2018Accelerated Methods for Deep Reinforcement Learningcited by 40
  14. 2019Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Realcited by 39
  15. 2018Multiagent Soft Q-Learningcited by 26
  16. 2020Macro-Action-Based Deep Multi-Agent Reinforcement Learningcited by 18
  17. 2020Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 13; 2024: 14; 2025: 17; 2026: 11.

68 papers.

Sort:

By area:

  1. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  2. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  3. 2023BenchMARL: Benchmarking Multi-Agent Reinforcement Learningcited by 31
  4. 2022PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaborationcited by 23
  5. 2022MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Librarycited by 21
  6. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
  7. 2022Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospectscited by 17
  8. 2022QGNN: Value Function Factorisation with Graph Neural Networkscited by 14
  9. 2023Aligning Language Models with Offline Learning from Human Feedbackcited by 14
  10. 2024FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learningcited by 13
  11. 2023Learning Cyber Defence Tactics from Scratch with Multi-Agent Reinforcement Learningcited by 12
  12. 2022PTDE: Personalized Training with Distilled Execution for Multi-Agent Reinforcement Learningcited by 11
38 more
  1. 2023From Explicit Communication to Tacit Cooperation:A Novel Paradigm for Cooperative MARLcited by 11
  2. 2025Iterative Self-Incentivization Empowers Large Language Models as Agentic Searcherscited by 11
  3. 2022MIXRTs: Toward Interpretable Multi-Agent Reinforcement Learning via Mixing Recurrent Soft Decision Treescited by 10
  4. 2023Self-Motivated Multi-Agent Explorationcited by 7
  5. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learningcited by 7
  6. 2024Variational Offline Multi-agent Skill Discoverycited by 6
  7. 2024IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learningcited by 6
  8. 2025Deep Meta Coordination Graphs for Multi-agent Reinforcement Learningcited by 6
  9. 2025QLLM: Do We Really Need a Mixing Network for Credit Assignment in Multi-Agent Reinforcement Learning?cited by 6
  10. 2026MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learningcited by 6
  11. 2022MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learningcited by 5
  12. 2023QuadSwarm: A Modular Multi-Quadrotor Simulator for Deep Reinforcement Learning with Direct Thrust Controlcited by 5
  13. 2024On Stateful Value Factorization in Multi-Agent Reinforcement Learningcited by 5
  14. 2024Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learningcited by 5
  15. 2024SMAC-Hard: Enabling Mixed Opponent Strategy Script and Self-play on SMACcited by 5
  16. 2023Best Possible Q-Learningcited by 4
  17. 2024Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learningcited by 4
  18. 2024Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learningcited by 4
  19. 2024GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARLcited by 3
  20. 2026Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learningcited by 3
  21. 2025TAG: A Decentralized Framework for Multi-Agent Hierarchical Reinforcement Learningcited by 2
  22. 2025Focusing Influence Mechanism for Multi-Agent Reinforcement Learningcited by 2
  23. 2025HLSMAC: A New StarCraft Multi-Agent Challenge for High-Level Strategic Decision-Makingcited by 2
  24. 2022A Policy Resonance Approach to Solve the Problem of Responsibility Diffusion in Multiagent Reinforcement Learningcited by 1
  25. 2022Learning Heterogeneous Agent Cooperation via Multiagent League Trainingcited by 1
  26. 2022CURO: Curriculum Learning for Relative Overgeneralizationcited by 1
  27. 2023How much can change in a year? Revisiting Evaluation in Multi-Agent Reinforcement Learningcited by 1
  28. 2024Self-Clustering Hierarchical Multi-Agent Reinforcement Learning with Extensible Cooperation Graphcited by 1
  29. 2024Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learningcited by 1
  30. 2025Beyond Shallow Behavior: Task-Efficient Value-Based Multi-Task Offline MARL via Skill Discoverycited by 1
  31. 2025Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Reviewcited by 1
  32. 2025GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspectivecited by 1
  33. 2025Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learningcited by 1
  34. 2022Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learningcited by 0
  35. 2023Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learningcited by 0
  36. 2023GHQ: Grouped Hybrid Q Learning for Heterogeneous Cooperative Multi-agent Reinforcement Learningcited by 0
  37. 2023QFree: A Universal Value Function Factorization for Multi-Agent Reinforcement Learningcited by 0
  38. 2024Mini Honor of Kings: A Lightweight Environment for Multi-Agent Reinforcement Learningcited by 0

50 most cited of 68 All 68 (JSON)