# Hierarchical Consensus-Based Multi-Agent Reinforcement Learning for Multi-Robot Cooperation Tasks

arXiv: https://arxiv.org/abs/2407.08164 · HTML: https://mlsanity.com/paper/2407.08164/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 27 of 27)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Multi-Agent Reinforcement Learning is a Sequence Modeling Problem](https://mlsanity.com/paper/2205.14953/) (2022)
- [Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.09870/) (2023)
- [Social NCE: Contrastive Learning of Socially-aware Motion Representations](https://mlsanity.com/paper/2012.11717/) (2020)
- [Deep Reinforcement Learning for Swarm Systems](https://mlsanity.com/paper/1807.06613/) (2018)
- [A Physics-Informed Deep Learning Paradigm for Traffic State and Fundamental Diagram Estimation](https://mlsanity.com/paper/2106.03142/) (2021)
- [Learning Actionable Representations from Visual Observations](https://mlsanity.com/paper/1808.00928/) (2018)
- [Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search](https://mlsanity.com/paper/1610.00673/) (2016)
- [Learning Structured Communication for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2002.04235/) (2020)
- [Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks](https://mlsanity.com/paper/2007.02991/) (2020)
- [Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game](https://mlsanity.com/paper/2305.12872/) (2023)
- [Leveraging Partial Symmetry for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2401.00167/) (2023)
- [ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.16186/) (2023)
- [GraphRARE: Reinforcement Learning Enhanced Graph Neural Network with Relative Entropy](https://mlsanity.com/paper/2312.09708/) (2023)
- [Consensus Learning for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.02583/) (2022)
- [Cooperative Multi-Agent Reinforcement Learning with Partial Observations](https://mlsanity.com/paper/2006.10822/) (2020)
- [Robust Multi-Agent Reinforcement Learning by Mutual Information Regularization](https://mlsanity.com/paper/2310.09833/) (2023)

## Built on by (most cited first, 13 of 13)

- [Internet of Agents: Fundamentals, Applications, and Challenges](https://mlsanity.com/paper/2505.07176/) (2025)
- [A Taxonomy of Hierarchical Multi-Agent Systems: Design Patterns, Coordination Mechanisms, and Industrial Applications](https://mlsanity.com/paper/2508.12683/) (2025)
- [HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution](https://mlsanity.com/paper/2604.17745/) (2026)
- [SIGMA: Sheaf-Informed Geometric Multi-Agent Pathfinding](https://mlsanity.com/paper/2502.06440/) (2025)
- [EmbodiedAgent: A Scalable Hierarchical Approach to Overcome Practical Challenge in Multi-Robot Control](https://mlsanity.com/paper/2504.10030/) (2025)
- [GlobeDiff: State Diffusion Process for Partial Observability in Multi-Agent Systems](https://mlsanity.com/paper/2602.15776/) (2026)
- [Spatial-Temporal Transformer with Curriculum Learning for EEG-Based Emotion Recognition](https://mlsanity.com/paper/2507.14698/) (2025)
- [Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.11824/) (2025)
- [Structured Diversity Control: A Dual-Level Framework for Group-Aware Multi-Agent Coordination](https://mlsanity.com/paper/2506.18651/) (2025)
- [Symmetry-Guided Multi-Agent Inverse Reinforcement Learning](https://mlsanity.com/paper/2509.08257/) (2025)
- [HCPO: Hierarchical Conductor-Based Policy Optimization in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2511.12123/) (2025)
- [Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuning](https://mlsanity.com/paper/2607.09422/) (2026)
- [Position: Beyond Training Time, Test-Time Coordination is Essential for Cooperative MARL](https://mlsanity.com/paper/2610.17409/) (2026)

## Abstract (arXiv)

In multi-agent reinforcement learning (MARL), the Centralized Training with Decentralized Execution (CTDE) framework is pivotal but struggles due to a gap: global state guidance in training versus reliance on local observations in execution, lacking global signals. Inspired by human societal consensus mechanisms, we introduce the Hierarchical Consensus-based Multi-Agent Reinforcement Learning (HC-MARL) framework to address this limitation. HC-MARL employs contrastive learning to foster a global consensus among agents, enabling cooperative behavior without direct communication. This approach enables agents to form a global consensus from local observations, using it as an additional piece of information to guide collaborative actions during execution. To cater to the dynamic requirements of various tasks, consensus is divided into multiple layers, encompassing both short-term and long-term considerations. Short-term observations prompt the creation of an immediate, low-layer consensus, while long-term observations contribute to the formation of a strategic, high-layer consensus. This process is further refined through an adaptive attention mechanism that dynamically adjusts the influence of each consensus layer. This mechanism optimizes the balance between immediate reactions and strategic planning, tailoring it to the specific demands of the task at hand. Extensive experiments and real-world applications in multi-robot systems showcase our framework's superior performance, marking significant advancements over baselines.