# Emergency action termination for immediate reaction in hierarchical reinforcement learning

arXiv: https://arxiv.org/abs/2211.06351 · HTML: https://mlsanity.com/paper/2211.06351/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Search on the Replay Buffer: Bridging Planning and Reinforcement Learning](https://mlsanity.com/paper/1906.05253/) (2019)
- [Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills](https://mlsanity.com/paper/2002.03647/) (2020)
- [Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?](https://mlsanity.com/paper/1909.10618/) (2019)
- [Hierarchical Reinforcement Learning By Discovering Intrinsic Options](https://mlsanity.com/paper/2101.06521/) (2021)
- [Learning Robot Skills with Temporal Variational Inference](https://mlsanity.com/paper/2006.16232/) (2020)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [Hierarchical Reinforcement Learning with Timed Subgoals](https://mlsanity.com/paper/2112.03100/) (2021)
- [Temporal-adaptive Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2002.02080/) (2020)
- [Learning to Interrupt: A Hierarchical Deep Reinforcement Learning Framework for Efficient Exploration](https://mlsanity.com/paper/1807.11150/) (2018)

## Abstract (arXiv)

Hierarchical decomposition of control is unavoidable in large dynamical systems. In reinforcement learning (RL), it is usually solved with subgoals defined at higher policy levels and achieved at lower policy levels. Reaching these goals can take a substantial amount of time, during which it is not verified whether they are still worth pursuing. However, due to the randomness of the environment, these goals may become obsolete. In this paper, we address this gap in the state-of-the-art approaches and propose a method in which the validity of higher-level actions (thus lower-level goals) is constantly verified at the higher level. If the actions, i.e. lower level goals, become inadequate, they are replaced by more appropriate ones. This way we combine the advantages of hierarchical RL, which is fast training, and flat RL, which is immediate reactivity. We study our approach experimentally on seven benchmark environments.