# The Termination Critic

arXiv: https://arxiv.org/abs/1902.09996 · HTML: https://mlsanity.com/paper/1902.09996/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2019.

## Built on by (most cited first, 24 of 24)

- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [TempoRL: Learning When to Act](https://mlsanity.com/paper/2106.05262/) (2021)
- [Generalised Policy Improvement with Geometric Policy Composition](https://mlsanity.com/paper/2206.08736/) (2022)
- [Temporally Abstract Partial Models](https://mlsanity.com/paper/2108.03213/) (2021)
- [MO2: Model-Based Offline Options](https://mlsanity.com/paper/2209.01947/) (2022)
- [Attention Option-Critic](https://mlsanity.com/paper/2201.02628/) (2022)
- [Selective Credit Assignment](https://mlsanity.com/paper/2202.09699/) (2022)
- [Toward Discovering Options that Achieve Faster Planning](https://mlsanity.com/paper/2205.12515/) (2022)
- [PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2306.06394/) (2023)
- [Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment](https://mlsanity.com/paper/2504.14805/) (2025)
- [The Paradox of Choice: Using Attention in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2201.09653/) (2022)
- [CRISP: Curriculum Inducing Primitive Informed Subgoal Prediction for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2304.03535/) (2023)
- [Iterative Option Discovery for Planning, by Planning](https://mlsanity.com/paper/2310.01569/) (2023)
- [Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robots](https://mlsanity.com/paper/2504.18794/) (2025)
- [Joint Learning of Hierarchical Neural Options and Abstract World Model](https://mlsanity.com/paper/2602.02799/) (2026)
- [AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning](https://mlsanity.com/paper/2605.06149/) (2026)
- [Reusable Options through Gradient-based Meta Learning](https://mlsanity.com/paper/2212.11726/) (2022)
- [Hierarchical Reinforcement Learning in Complex 3D Environments](https://mlsanity.com/paper/2302.14451/) (2023)
- [Accelerating Task Generalisation with Multi-Level Skill Hierarchies](https://mlsanity.com/paper/2411.02998/) (2024)
- [Select before Act: Spatially Decoupled Action Repetition for Continuous Control](https://mlsanity.com/paper/2502.06919/) (2025)
- [Autonomous Curriculum Design via Relative Entropy Based Task Modifications](https://mlsanity.com/paper/2502.21166/) (2025)
- [Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPs](https://mlsanity.com/paper/2507.16473/) (2025)
- [Enhancing Hierarchical Reinforcement Learning through Change Point Detection in Time Series](https://mlsanity.com/paper/2510.24988/) (2025)
- [When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Critic](https://mlsanity.com/paper/2609.05508/) (2026)

## Abstract (arXiv)

In this work, we consider the problem of autonomously discovering behavioral abstractions, or options, for reinforcement learning agents. We propose an algorithm that focuses on the termination condition, as opposed to -- as is common -- the policy. The termination condition is usually trained to optimize a control objective: an option ought to terminate if another has better value. We offer a different, information-theoretic perspective, and propose that terminations should focus instead on the compressibility of the option's encoding -- arguably a key reason for using abstractions. To achieve this algorithmically, we leverage the classical options framework, and learn the option transition model as a "critic" for the termination condition. Using this model, we derive gradients that optimize the desired criteria. We show that the resulting options are non-trivial, intuitively meaningful, and useful for learning and planning.