Hierarchical Strategies for Cooperative Multi-Agent Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Adequate strategizing of agents behaviors is essential to solving cooperative MARL problems. One intuitively beneficial yet uncommon method in this domain is predicting agents future behaviors and planning accordingly. Leveraging this point, we propose a two-level hierarchical architecture that combines a novel information-theoretic objective with a trajectory prediction model to learn a strategy. To this end, we introduce a latent policy that learns two types of latent strategies: individual zA, and relational zR using a modified Graph Attention Network module to extract interaction features. We encourage each agent to behave according to the strategy by conditioning its local Q functions on zA, and we further equip agents with a shared Q function that conditions on zR. Additionally, we introduce two regularizers to allow predicted trajectories to be accurate and rewarding. Empirical results on Google Research Football (GRF) and StarCraft (SC) II micromanagement tasks show that our method establishes a new state of the art being, to the best of our knowledge, the first MARL algorithm to solve all super hard SC II scenarios as well as the GRF full game with a win rate higher than 95%, thus outperforming all existing methods. Videos and brief overview of the methods and results are available at: https://sites.google.com/view/hier-strats-marl/home.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 52 of its 91 references that are in the map. First posted on arXiv 14 Dec 2022.
Builds on
52 of 91 references.
References · 52
- 2017Graph Attention Networkscited by 8,854
- 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
- 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
- 2016Deep Variational Information Bottleneckcited by 915
- 2016Conditional Image Generation with PixelCNN Decoderscited by 859
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
- 2017Counterfactual Multi-Agent Policy Gradientscited by 733
- 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
- 2019The StarCraft Multi-Agent Challengecited by 589
- 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
- 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
- 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
38 more
- 2018Neural Relational Inference for Interacting Systemscited by 339
- 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
- 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
- 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
- 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
- 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
- 2019MAVEN: Multi-Agent Variational Explorationcited by 181
- 2019Multiple Futures Predictioncited by 178
- 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
- 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
- 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
- 2020ROMA: Multi-Agent Reinforcement Learning with Emergent Rolescited by 135
- 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
- 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
- 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
- 2019Deep Coordination Graphscited by 96
- 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 96
- 2019Learning Nearly Decomposable Value Functions Via Communication Minimizationcited by 85
- 2020Qatten: A General Framework for Cooperative Multiagent Reinforcement Learningcited by 81
- 2019Influence-Based Multi-Agent Explorationcited by 75
- 2019Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learningcited by 68
- 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
- 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
- 2017Z-Forcing: Training Stochastic Recurrent Networkscited by 51
- 2020Social diversity and social preferences in mixed-motive reinforcement learningcited by 49
- 2021Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Explorationcited by 49
- 2020Value-Decomposition Multi-Agent Actor-Criticscited by 47
- 2018Generating Multi-Agent Trajectories using Programmatic Weak Supervisioncited by 46
- 2017Guided Deep Reinforcement Learning for Swarm Systemscited by 44
- 2019Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discoverycited by 42
- 2021Hierarchical Skills for Efficient Explorationcited by 37
- 2019Multi-Agent Actor-Critic with Hierarchical Graph Attention Networkcited by 33
- 2021Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Compositioncited by 28
- 2020The Emergence of Individualitycited by 22
- 2019Multi-Agent Reinforcement Learning with Multi-Step Generative Modelscited by 18
- 2020Multi-agent Trajectory Prediction with Fuzzy Query Attentioncited by 16
- 2018The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spacescited by 15
- 2017Prediction and Control with Temporal Segment Modelscited by 14
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.