# Multi-Agent Actor-Critic with Hierarchical Graph Attention Network

arXiv: https://arxiv.org/abs/1909.12557 · HTML: https://mlsanity.com/paper/1909.12557/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2019.

## Built on by (most cited first, 30 of 33)

- [Hierarchical Learning for Generation with Long Source Sequences](https://mlsanity.com/paper/2104.07545/) (2021)
- [Graph Neural Networks in IoT: A Survey](https://mlsanity.com/paper/2203.15935/) (2022)
- [Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts](https://mlsanity.com/paper/2105.03363/) (2021)
- [Learning Efficient Multi-Agent Cooperative Visual Exploration](https://mlsanity.com/paper/2110.05734/) (2021)
- [Generalization in Cooperative Multi-Agent Systems](https://mlsanity.com/paper/2202.00104/) (2022)
- [Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Exploration](https://mlsanity.com/paper/2301.03398/) (2023)
- [Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models](https://mlsanity.com/paper/2406.15836/) (2024)
- [Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.04612/) (2022)
- [Efficient Multi-agent Reinforcement Learning by Planning](https://mlsanity.com/paper/2405.11778/) (2024)
- [Interaction Pattern Disentangling for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2207.03902/) (2022)
- [The Importance of Credo in Multiagent Learning](https://mlsanity.com/paper/2204.07471/) (2022)
- [MDPGT: Momentum-based Decentralized Policy Gradient Tracking](https://mlsanity.com/paper/2112.02813/) (2021)
- [Learning Graph-Enhanced Commander-Executor for Multi-Agent Navigation](https://mlsanity.com/paper/2302.04094/) (2023)
- [Distributed Learning over Networks with Graph-Attention-Based Personalization](https://mlsanity.com/paper/2305.13041/) (2023)
- [Cooperative and Competitive Biases for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2101.06890/) (2021)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [A Graph Attention Learning Approach to Antenna Tilt Optimization](https://mlsanity.com/paper/2112.14843/) (2021)
- [S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?](https://mlsanity.com/paper/2206.11054/) (2022)
- [Deep Reinforcement Learning based Autonomous Decision-Making for Cooperative UAVs: A Search and Rescue Real World Application](https://mlsanity.com/paper/2502.20326/) (2025)
- [Trust-MARL: Trust-Based Multi-Agent Reinforcement Learning Framework for Cooperative On-Ramp Merging Control in Heterogeneous Traffic Flow](https://mlsanity.com/paper/2506.12600/) (2025)
- [Graph Attention-based Reinforcement Learning for Trajectory Design and Resource Assignment in Multi-UAV Assisted Communication](https://mlsanity.com/paper/2401.17880/) (2024)
- [Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2507.23604/) (2025)
- [Soft Hierarchical Graph Recurrent Networks for Many-Agent Partially Observable Environments](https://mlsanity.com/paper/2109.02032/) (2021)
- [Distributed Multi-Agent Deep Reinforcement Learning for Robust Coordination against Noise](https://mlsanity.com/paper/2205.09705/) (2022)
- [A Policy Resonance Approach to Solve the Problem of Responsibility Diffusion in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2208.07753/) (2022)
- [Hierarchical Strategies for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.07397/) (2022)
- [Graph Attention with Hierarchies for Multi-hop Question Answering](https://mlsanity.com/paper/2301.11792/) (2023)
- [Active Neural Topological Mapping for Multi-Agent Exploration](https://mlsanity.com/paper/2311.00252/) (2023)
- [Towards Fault Tolerance in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2412.00534/) (2024)
- [HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2606.29126/) (2026)

## Abstract (arXiv)

Most previous studies on multi-agent reinforcement learning focus on deriving decentralized and cooperative policies to maximize a common reward and rarely consider the transferability of trained policies to new tasks. This prevents such policies from being applied to more complex multi-agent tasks. To resolve these limitations, we propose a model that conducts both representation learning for multiple agents using hierarchical graph attention network and policy learning using multi-agent actor-critic. The hierarchical graph attention network is specially designed to model the hierarchical relationships among multiple agents that either cooperate or compete with each other to derive more advanced strategic policies. Two attention networks, the inter-agent and inter-group attention layers, are used to effectively model individual and group level interactions, respectively. The two attention networks have been proven to facilitate the transfer of learned policies to new tasks with different agent compositions and allow one to interpret the learned strategies. Empirically, we demonstrate that the proposed model outperforms existing methods in several mixed cooperative and competitive tasks.