# Learnings Options End-to-End for Continuous Action Tasks

arXiv: https://arxiv.org/abs/1712.00004 · HTML: https://mlsanity.com/paper/1712.00004/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2017.

## Built on by (most cited first, 24 of 24)

- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [Hierarchical Reinforcement Learning By Discovering Intrinsic Options](https://mlsanity.com/paper/2101.06521/) (2021)
- [Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning](https://mlsanity.com/paper/2111.03189/) (2021)
- [Flexible Option Learning](https://mlsanity.com/paper/2112.03097/) (2021)
- [Strategy and Skill Learning for Physics-based Table Tennis Animation](https://mlsanity.com/paper/2407.16210/) (2024)
- [HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents](https://mlsanity.com/paper/2602.16165/) (2026)
- [Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration](https://mlsanity.com/paper/2211.04786/) (2022)
- [Scalable Option Learning in High-Throughput Environments](https://mlsanity.com/paper/2509.00338/) (2025)
- [oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions](https://mlsanity.com/paper/2002.09043/) (2020)
- [PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2306.06394/) (2023)
- [PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling](https://mlsanity.com/paper/2404.13423/) (2024)
- [CRISP: Curriculum Inducing Primitive Informed Subgoal Prediction for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2304.03535/) (2023)
- [DIPPER: Direct Preference Optimization to Accelerate Primitive-Enabled Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2406.10892/) (2024)
- [SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environments](https://mlsanity.com/paper/2407.18913/) (2024)
- [Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robots](https://mlsanity.com/paper/2504.18794/) (2025)
- [Electric Bus Charging Schedules Relying on Real Data-Driven Targets Based on Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/2505.10262/) (2025)
- [Interpretable Reinforcement Learning Inspired by Piaget's Theory of Cognitive Development](https://mlsanity.com/paper/2102.00572/) (2021)
- [Coordinating Policies Among Multiple Agents via an Intelligent Communication Channel](https://mlsanity.com/paper/2205.10607/) (2022)
- [Accelerating Task Generalisation with Multi-Level Skill Hierarchies](https://mlsanity.com/paper/2411.02998/) (2024)
- [Enter the Void - Planning to Seek Entropy When Reward is Scarce](https://mlsanity.com/paper/2505.16787/) (2025)
- [Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPs](https://mlsanity.com/paper/2507.16473/) (2025)
- [Enabling Option Learning in Sparse Rewards with Hindsight Experience Replay](https://mlsanity.com/paper/2602.13865/) (2026)
- [Hierarchical Behaviour Spaces](https://mlsanity.com/paper/2604.24558/) (2026)
- [Hierarchical Reinforcement Learning for Sparse-Reward Search in Commutative Algebra](https://mlsanity.com/paper/2606.22922/) (2026)

## Abstract (arXiv)

We present new results on learning temporally extended actions for continuoustasks, using the options framework (Suttonet al.[1999b], Precup [2000]). In orderto achieve this goal we work with the option-critic architecture (Baconet al.[2017])using a deliberation cost and train it with proximal policy optimization (Schulmanet al.[2017]) instead of vanilla policy gradient. Results on Mujoco domains arepromising, but lead to interesting questions aboutwhena given option should beused, an issue directly connected to the use of initiation sets.