# Multi-agent Hierarchical Reinforcement Learning with Dynamic Termination

arXiv: https://arxiv.org/abs/1910.09508 · HTML: https://mlsanity.com/paper/1910.09508/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2019.

## Built on by (most cited first, 6 of 6)

- [Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.10113/) (2022)
- [Guiding Multi-agent Multi-task Reinforcement Learning by a Hierarchical Framework with Logical Reward Shaping](https://mlsanity.com/paper/2411.01184/) (2024)
- [Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2209.10003/) (2022)
- [Hierarchically Structured Scheduling and Execution of Tasks in a Multi-Agent Environment](https://mlsanity.com/paper/2203.03021/) (2022)
- [Asynchronous, Option-Based Multi-Agent Policy Gradient: A Conditional Reasoning Approach](https://mlsanity.com/paper/2203.15925/) (2022)
- [Coordinating Policies Among Multiple Agents via an Intelligent Communication Channel](https://mlsanity.com/paper/2205.10607/) (2022)

## Abstract (arXiv)

In a multi-agent system, an agent's optimal policy will typically depend on the policies chosen by others. Therefore, a key issue in multi-agent systems research is that of predicting the behaviours of others, and responding promptly to changes in such behaviours. One obvious possibility is for each agent to broadcast their current intention, for example, the currently executed option in a hierarchical reinforcement learning framework. However, this approach results in inflexibility of agents if options have an extended duration and are dynamic. While adjusting the executed option at each step improves flexibility from a single-agent perspective, frequent changes in options can induce inconsistency between an agent's actual behaviour and its broadcast intention. In order to balance flexibility and predictability, we propose a dynamic termination Bellman equation that allows the agents to flexibly terminate their options. We evaluate our model empirically on a set of multi-agent pursuit and taxi tasks, and show that our agents learn to adapt flexibly across scenarios that require different termination behaviours.