mlsanityData to 7 Oct 2026

Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

arXiv 2006.10800 ↗First posted 18 Jun 2020cs.LG cs.MA stat.MLcited by 197

Not on the map. Why?

Abstract (arXiv)

QMIX is a popular Q-learning algorithm for cooperative MARL in the centralised training and decentralised execution paradigm. In order to enable easy decentralisation, QMIX restricts the joint action Q-values it can represent to be a monotonic mixing of each agent's utilities. However, this restriction prevents it from representing value functions in which an agent's ordering over its actions can depend on other agents' actions. To analyse this representational limitation, we first formalise the objective QMIX optimises, which allows us to view QMIX as an operator that first computes the Q-learning targets and then projects them into the space representable by QMIX. This projection returns a representable Q-value that minimises the unweighted squared error across all joint actions. We show in particular that this projection can fail to recover the optimal policy even with access to Q^*, which primarily stems from the equal weighting placed on each joint action. We rectify this by introducing a weighting into the projection, in order to place more importance on the better joint actions. We propose two weighting schemes and prove that they recover the correct maximal action for any joint action Q-values, and therefore for Q^* as well. Based on our analysis and results in the tabular setting, we introduce two scalable versions of our algorithm, Centrally-Weighted (CW) QMIX and Optimistically-Weighted (OW) QMIX and demonstrate improved performance on both predator-prey and challenging multi-agent StarCraft benchmark tasks.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 43; 2023: 36; 2024: 28; 2025: 37; 2026: 24.

197 papers.

Sort:

By area:

  1. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  2. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  3. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 96
  4. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  5. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  6. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  7. 2021Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learningcited by 54
  8. 2021Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Explorationcited by 49
  9. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  10. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 46
  11. 2021Deep Multiagent Reinforcement Learning: Challenges and Directionscited by 42
  12. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
38 more
  1. 2021Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Surveycited by 34
  2. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  3. 2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
  4. 2021On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)cited by 31
  5. 2021Context-Aware Sparse Deep Coordination Graphscited by 27
  6. 2021SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learningcited by 26
  7. 2021Unbiased Asymmetric Reinforcement Learning under Partial Observabilitycited by 24
  8. 2022What is the Solution for State-Adversarial Multi-Agent Reinforcement Learning?cited by 21
  9. 2023Measuring Acoustics with Collaborative Multiple Agentscited by 21
  10. 2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
  11. 2021Regularized Softmax Deep Multi-Agent Q-Learningcited by 19
  12. 2021HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanismcited by 19
  13. 2022More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorizationcited by 19
  14. 2022A Survey on Reinforcement Learning in Aviation Applicationscited by 19
  15. 2022Contrastive Identity-Aware Learning for Multi-Agent Value Decompositioncited by 19
  16. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
  17. 2023Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?cited by 18
  18. 2021Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Trainingcited by 16
  19. 2022ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependencycited by 15
  20. 2023Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilitiescited by 15
  21. 2021Self-Organized Polynomial-Time Coordination Graphscited by 14
  22. 2022Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradientscited by 12
  23. 2022A Survey on Reinforcement Learning Methods in Character Animationcited by 12
  24. 2022A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learningcited by 12
  25. 2025Enhancing Multi-Agent Systems via Reinforcement Learning with LLM-based Planner and Graph-based Policycited by 12
  26. 2025Multi-Agent Reinforcement Learning for Adaptive Resource Orchestration in Cloud-Native Clusterscited by 12
  27. 2023Informed POMDP: Leveraging Additional Information in Model-Based RLcited by 11
  28. 2021Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid Worldcited by 10
  29. 2026Counterfactual Credit Policy Optimization for Multi-Agent Collaborationcited by 10
  30. 2021LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammatescited by 9
  31. 2023TransfQMix: Transformers for Leveraging the Graph Structure of Multi-Agent Reinforcement Learning Problemscited by 9
  32. 2023IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARLcited by 9
  33. 2023Scalable Multi-agent Covering Option Discovery based on Kronecker Graphscited by 9
  34. 2024On Centralized Critics in Multi-Agent Reinforcement Learningcited by 9
  35. 2022Sparse Adversarial Attack in Multi-agent Reinforcement Learningcited by 8
  36. 2022Consensus Learning for Cooperative Multi-Agent Reinforcement Learningcited by 8
  37. 2022Interaction Pattern Disentangling for Multi-Agent Reinforcement Learningcited by 8
  38. 2023Mean-Field Control based Approximation of Multi-Agent Reinforcement Learning in Presence of a Non-decomposable Shared Global Statecited by 8

50 most cited of 197 All 197 (JSON)