# A Laplacian Framework for Option Discovery in Reinforcement Learning

arXiv: https://arxiv.org/abs/1703.00956 · HTML: https://mlsanity.com/paper/1703.00956/

## Key facts

- Cited by 110 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2017.

## Built on by (most cited first, 30 of 110)

- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [HIQL: Offline Goal-Conditioned RL with Latent States as Actions](https://mlsanity.com/paper/2307.11949/) (2023)
- [Motif: Intrinsic Motivation from Artificial Intelligence Feedback](https://mlsanity.com/paper/2310.00166/) (2023)
- [Foundation Policies with Hilbert Representations](https://mlsanity.com/paper/2402.15567/) (2024)
- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [Does Zero-Shot Reinforcement Learning Exist?](https://mlsanity.com/paper/2209.14935/) (2022)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [Lipschitz-constrained Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00914/) (2022)
- [On The Effect of Auxiliary Tasks on Representation Dynamics](https://mlsanity.com/paper/2102.13089/) (2021)
- [Temporal Abstraction in Reinforcement Learning with the Successor Representation](https://mlsanity.com/paper/2110.05740/) (2021)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [Spectral Decomposition Representation for Reinforcement Learning](https://mlsanity.com/paper/2208.09515/) (2022)
- [Learning to Play with Intrinsically-Motivated Self-Aware Agents](https://mlsanity.com/paper/1802.07442/) (2018)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- [Deep Laplacian-based Options for Temporally-Extended Exploration](https://mlsanity.com/paper/2301.11181/) (2023)
- [On the Generalization of Representations in Reinforcement Learning](https://mlsanity.com/paper/2203.00543/) (2022)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [Metrics and continuity in reinforcement learning](https://mlsanity.com/paper/2102.01514/) (2021)
- [Flexible Option Learning](https://mlsanity.com/paper/2112.03097/) (2021)
- [MaestroMotif: Skill Design from Artificial Intelligence Feedback](https://mlsanity.com/paper/2412.08542/) (2024)
- [Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing](https://mlsanity.com/paper/2107.05545/) (2021)
- [Intelligent problem-solving as integrated hierarchical reinforcement learning](https://mlsanity.com/paper/2208.08731/) (2022)
- [Proto Successor Measure: Representing the Behavior Space of an RL Agent](https://mlsanity.com/paper/2411.19418/) (2024)
- [Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching](https://mlsanity.com/paper/2110.14457/) (2021)
- [Reward-Respecting Subtasks for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2202.03466/) (2022)
- [Unsupervised Learning of Temporal Abstractions with Slot-based Transformers](https://mlsanity.com/paper/2203.13573/) (2022)
- [Learning Dynamics and Generalization in Reinforcement Learning](https://mlsanity.com/paper/2206.02126/) (2022)
- [Generalised Policy Improvement with Geometric Policy Composition](https://mlsanity.com/paper/2206.08736/) (2022)

## Abstract (arXiv)

Representation learning and option discovery are two of the biggest challenges in reinforcement learning (RL). Proto-value functions (PVFs) are a well-known approach for representation learning in MDPs. In this paper we address the option discovery problem by showing how PVFs implicitly define options. We do it by introducing eigenpurposes, intrinsic reward functions derived from the learned representations. The options discovered from eigenpurposes traverse the principal directions of the state space. They are useful for multiple tasks because they are discovered without taking the environment's rewards into consideration. Moreover, different options act at different time scales, making them helpful for exploration. We demonstrate features of eigenpurposes in traditional tabular domains as well as in Atari 2600 games.