Hierarchical Task Network Planning for Facilitating Cooperative Multi-Agent Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Exploring sparse reward multi-agent reinforcement learning (MARL) environments with traps in a collaborative manner is a complex task. Agents typically fail to reach the goal state and fall into traps, which affects the overall performance of the system. To overcome this issue, we present SOMARL, a framework that uses prior knowledge to reduce the exploration space and assist learning. In SOMARL, agents are treated as part of the MARL environment, and symbolic knowledge is embedded using a tree structure to build a knowledge hierarchy. The framework has a two-layer hierarchical structure, comprising a hybrid module with a Hierarchical Task Network (HTN) planning and meta-controller at the higher level, and a MARL-based interactive module at the lower level. The HTN module and meta-controller use Hierarchical Domain Definition Language (HDDL) and the option framework to formalize symbolic knowledge and obtain domain knowledge and a symbolic option set, respectively. Moreover, the HTN module leverages domain knowledge to guide low-level agent exploration by assisting the meta-controller in selecting symbolic options. The meta-controller further computes intrinsic rewards of symbolic options to limit exploration behavior and adjust HTN planning solutions as needed. We evaluate SOMARL on two benchmarks, FindTreasure and MoveBox, and report superior performance over state-of-the-art MARL and subgoal-based baselines for MARL environments significantly.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 42 references that are in the map. First posted on arXiv 14 Jun 2023.
Builds on
23 of 42 references.
References · 23
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
- 2017Hindsight Experience Replaycited by 1,096
- 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
- 2017Counterfactual Multi-Agent Policy Gradientscited by 733
- 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
- 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
- 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
- 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
- 2016Learning Multiagent Communication with Backpropagationcited by 355
- 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
11 more
- 2019MAVEN: Multi-Agent Variational Explorationcited by 181
- 2019A Review of Cooperative Multi-Agent Deep Reinforcement Learningcited by 147
- 2019Multi-Agent Game Abstraction via Graph Attention Neural Networkcited by 81
- 2018SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planningcited by 77
- 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 55
- 2018PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Makingcited by 51
- 2019Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discoverycited by 42
- 2019From Few to More: Large-scale Dynamic Multiagent Curriculum Learningcited by 36
- 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
- 2018Managing engineering systems with large state and action spaces through deep reinforcement learningcited by 32
- 2018Discovering Underlying Plans Based on Shallow Modelscited by 9
Built on by
Citing papers per year: 2026: 1.
1 paper.