mlsanityData to 7 Oct 2026

HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism

arXiv 2110.07246 ↗First posted 14 Oct 2021cs.MA cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Recently, some challenging tasks in multi-agent systems have been solved by some hierarchical reinforcement learning methods. Inspired by the intra-level and inter-level coordination in the human nervous system, we propose a novel value decomposition framework HAVEN based on hierarchical reinforcement learning for fully cooperative multi-agent problems. To address the instability arising from the concurrent optimization of policies between various levels and agents, we introduce the dual coordination mechanism of inter-level and inter-agent strategies by designing reward functions in a two-level hierarchy. HAVEN does not require domain knowledge and pre-training, and can be applied to any value decomposition variant. Our method achieves desirable results on different decentralized partially observable Markov decision process domains and outperforms other popular multi-agent hierarchical reinforcement learning algorithms.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 30 of its 45 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

30 of 45 references.

References · 30

  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  2. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  3. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  4. 2019The StarCraft Multi-Agent Challengecited by 589
  5. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  6. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  7. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  8. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  9. 2016The Option-Critic Architecturecited by 433
  10. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  11. 2016Learning Multiagent Communication with Backpropagationcited by 355
  12. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
18 more
  1. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  2. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  3. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  4. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  5. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  6. 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
  7. 2017Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Gamescited by 93
  8. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 57
  9. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  10. 2020Value-Decomposition Multi-Agent Actor-Criticscited by 47
  11. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  12. 2019Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discoverycited by 42
  13. 2019Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewardscited by 38
  14. 2019Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learningcited by 32
  15. 2018Learning Representations in Model-Free Hierarchical Reinforcement Learningcited by 13
  16. 2018Reinforcement Learning based Recommender System using Biclustering Techniquecited by 12
  17. 2019Inter-Level Cooperation in Hierarchical Reinforcement Learningcited by 7
  18. 2021SIDE: State Inference for Partially Observable Cooperative Multi-Agent Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 2; 2024: 3; 2025: 8; 2026: 3.

19 papers.

Sort:

By area:

  1. 2022Asynchronous Actor-Critic for Multi-Agent Reinforcement Learningcited by 16
  2. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  3. 2022Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learningcited by 6
  4. 2024Variational Offline Multi-agent Skill Discoverycited by 6
  5. 2024Multi-Agent Reinforcement Learning with Hierarchical Coordination for Emergency Responder Stationingcited by 4
  6. 2025Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systemscited by 4
  7. 2022Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observabilitycited by 2
  8. 2023SEA: A Spatially Explicit Architecture for Multi-Agent Reinforcement Learningcited by 2
  9. 2024Subequivariant Reinforcement Learning in 3D Multi-Entity Physical Environmentscited by 2
  10. 2025Enhancing Sample Efficiency in Multi-Agent RL with Uncertainty Quantification and Selective Explorationcited by 1
  11. 2025Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning via Incorporating Generalized Human Expertisecited by 1
  12. 2025Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learningcited by 1
7 more
  1. 2025Optimization without Future Compromises? Decentralized Coordination via Collective and Reinforcement Learningcited by 1
  2. 2026SyncPlan: Long-Horizon LLM Coordination with Explicit Synchronization and Adaptive Correctioncited by 1
  3. 2025Zero-Incentive Dynamics: a look at reward sparsity through the lens of unrewarded subgoalscited by 0
  4. 2025HCPO: Hierarchical Conductor-Based Policy Optimization in Multi-Agent Reinforcement Learningcited by 0
  5. 2025Conditional Diffusion Model for Multi-Agent Dynamic Task Decompositioncited by 0
  6. 2026Learning to Reflect: Hierarchical Multi-Agent Reinforcement Learning for CSI-Free mmWave Beam-Focusingcited by 0
  7. 2026Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Controlcited by 0