# When Waiting is not an Option : Learning Options with a Deliberation Cost

arXiv: https://arxiv.org/abs/1709.04571 · HTML: https://mlsanity.com/paper/1709.04571/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2017.

## Built on by (most cited first, 30 of 58)

- [Code as Reward: Empowering Reinforcement Learning with VLMs](https://mlsanity.com/paper/2402.04764/) (2024)
- [Deep Laplacian-based Options for Temporally-Extended Exploration](https://mlsanity.com/paper/2301.11181/) (2023)
- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [TempoRL: Learning When to Act](https://mlsanity.com/paper/2106.05262/) (2021)
- [Flexible Option Learning](https://mlsanity.com/paper/2112.03097/) (2021)
- [MaestroMotif: Skill Design from Artificial Intelligence Feedback](https://mlsanity.com/paper/2412.08542/) (2024)
- [Deciding What to Learn: A Rate-Distortion Approach](https://mlsanity.com/paper/2101.06197/) (2021)
- [Learning Options via Compression](https://mlsanity.com/paper/2212.04590/) (2022)
- [Hierarchical Reinforcement Learning with Timed Subgoals](https://mlsanity.com/paper/2112.03100/) (2021)
- [HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism](https://mlsanity.com/paper/2110.07246/) (2021)
- [Humans decompose tasks by trading off utility and computational cost](https://mlsanity.com/paper/2211.03890/) (2022)
- [Reward-Respecting Subtasks for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2202.03466/) (2022)
- [Generalised Policy Improvement with Geometric Policy Composition](https://mlsanity.com/paper/2206.08736/) (2022)
- [Dynamic Decision Frequency with Continuous Options](https://mlsanity.com/paper/2212.04407/) (2022)
- [Temporally Abstract Partial Models](https://mlsanity.com/paper/2108.03213/) (2021)
- [Hierarchical Reinforcement Learning for Power Network Topology Control](https://mlsanity.com/paper/2311.02129/) (2023)
- [MO2: Model-Based Offline Options](https://mlsanity.com/paper/2209.01947/) (2022)
- [Scalable Multi-agent Covering Option Discovery based on Kronecker Graphs](https://mlsanity.com/paper/2307.11629/) (2023)
- [When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL](https://mlsanity.com/paper/2406.01163/) (2024)
- [SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions](https://mlsanity.com/paper/2410.18416/) (2024)
- [Attention Option-Critic](https://mlsanity.com/paper/2201.02628/) (2022)
- [Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration](https://mlsanity.com/paper/2211.04786/) (2022)
- [Granger Causal Interaction Skill Chains](https://mlsanity.com/paper/2306.09509/) (2023)
- [Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems](https://mlsanity.com/paper/2507.14850/) (2025)
- [Scalable Option Learning in High-Throughput Environments](https://mlsanity.com/paper/2509.00338/) (2025)
- [Learning Multi-agent Skills for Tabular Reinforcement Learning using Factor Graphs](https://mlsanity.com/paper/2201.08227/) (2022)
- [Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning](https://mlsanity.com/paper/2201.08115/) (2022)
- [Toward Discovering Options that Achieve Faster Planning](https://mlsanity.com/paper/2205.12515/) (2022)
- [PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2306.06394/) (2023)
- [PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling](https://mlsanity.com/paper/2404.13423/) (2024)

## Abstract (arXiv)

Recent work has shown that temporally extended actions (options) can be learned fully end-to-end as opposed to being specified in advance. While the problem of "how" to learn options is increasingly well understood, the question of "what" good options should be has remained elusive. We formulate our answer to what "good" options should be in the bounded rationality framework (Simon, 1957) through the notion of deliberation cost. We then derive practical gradient-based learning algorithms to implement this objective. Our results in the Arcade Learning Environment (ALE) show increased performance and interpretability.