# Successor Options: An Option Discovery Framework for Reinforcement Learning

arXiv: https://arxiv.org/abs/1905.05731 · HTML: https://mlsanity.com/paper/1905.05731/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2019.

## Built on by (most cited first, 23 of 23)

- [Temporal Abstraction in Reinforcement Learning with the Successor Representation](https://mlsanity.com/paper/2110.05740/) (2021)
- [Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2111.09858/) (2021)
- [Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2505.12737/) (2025)
- [Exploring the limits of Hierarchical World Models in Reinforcement Learning](https://mlsanity.com/paper/2406.00483/) (2024)
- [MO2: Model-Based Offline Options](https://mlsanity.com/paper/2209.01947/) (2022)
- [Local Explanations for Reinforcement Learning](https://mlsanity.com/paper/2202.03597/) (2022)
- [Creating Multi-Level Skill Hierarchies in Reinforcement Learning](https://mlsanity.com/paper/2306.09980/) (2023)
- [Predictive representations: building blocks of intelligence](https://mlsanity.com/paper/2402.06590/) (2024)
- [Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning](https://mlsanity.com/paper/2412.16395/) (2024)
- [Option Discovery Using LLM-guided Semantic Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2503.19007/) (2025)
- [Deep Reinforcement Learning Based Navigation with Macro Actions and Topological Maps](https://mlsanity.com/paper/2504.18300/) (2025)
- [Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction](https://mlsanity.com/paper/2408.11816/) (2024)
- [Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learning](https://mlsanity.com/paper/2605.29782/) (2026)
- [Option Transfer and SMDP Abstraction with Successor Features](https://mlsanity.com/paper/2110.09196/) (2021)
- [Discrete State-Action Abstraction via the Successor Representation](https://mlsanity.com/paper/2206.03467/) (2022)
- [Subgoal Discovery Using a Free Energy Paradigm and State Aggregations](https://mlsanity.com/paper/2412.16687/) (2024)
- [Landmark-Assisted Monte Carlo Planning](https://mlsanity.com/paper/2508.11493/) (2025)
- [Accelerating Task Generalisation with Multi-Level Skill Hierarchies](https://mlsanity.com/paper/2411.02998/) (2024)
- [Improving planning and MBRL with temporally-extended actions](https://mlsanity.com/paper/2505.15754/) (2025)
- [Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2512.04958/) (2025)
- [Intrinsic Reward Policy Optimization for Sparse-Reward Environments](https://mlsanity.com/paper/2601.21391/) (2026)
- [Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning](https://mlsanity.com/paper/2605.13207/) (2026)
- [Structure-Induced Information for Rerooting Levin Tree Search](https://mlsanity.com/paper/2605.30664/) (2026)

## Abstract (arXiv)

The options framework in reinforcement learning models the notion of a skill or a temporally extended sequence of actions. The discovery of a reusable set of skills has typically entailed building options, that navigate to bottleneck states. This work adopts a complementary approach, where we attempt to discover options that navigate to landmark states. These states are prototypical representatives of well-connected regions and can hence access the associated region with relative ease. In this work, we propose Successor Options, which leverages Successor Representations to build a model of the state space. The intra-option policies are learnt using a novel pseudo-reward and the model scales to high-dimensional spaces easily. Additionally, we also propose an Incremental Successor Options model that iterates between constructing Successor Representations and building options, which is useful when robust Successor Representations cannot be built solely from primitive actions. We demonstrate the efficacy of our approach on a collection of grid-worlds, and on the high-dimensional robotic control environment of Fetch.