# Deep Laplacian-based Options for Temporally-Extended Exploration

arXiv: https://arxiv.org/abs/2301.11181 · HTML: https://mlsanity.com/paper/2301.11181/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 65 references that are in the map.
- First posted on arXiv 26 Jan 2023.

## Builds on (most cited first, 30 of 35)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [A Laplacian Framework for Option Discovery in Reinforcement Learning](https://mlsanity.com/paper/1703.00956/) (2017)
- [Eigenoption Discovery through the Deep Successor Representation](https://mlsanity.com/paper/1710.11089/) (2017)
- [The Laplacian in RL: Learning Representations with Efficient Approximations](https://mlsanity.com/paper/1810.04586/) (2018)
- [Does Zero-Shot Reinforcement Learning Exist?](https://mlsanity.com/paper/2209.14935/) (2022)
- [CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00161/) (2022)
- [Lipschitz-constrained Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00914/) (2022)
- [When Waiting is not an Option : Learning Options with a Deliberation Cost](https://mlsanity.com/paper/1709.04571/) (2017)
- [On The Effect of Auxiliary Tasks on Representation Dynamics](https://mlsanity.com/paper/2102.13089/) (2021)
- [Temporal Abstraction in Reinforcement Learning with the Successor Representation](https://mlsanity.com/paper/2110.05740/) (2021)
- [Spectral Decomposition Representation for Reinforcement Learning](https://mlsanity.com/paper/2208.09515/) (2022)
- [Spectral Inference Networks: Unifying Deep and Spectral Learning](https://mlsanity.com/paper/1806.02215/) (2018)
- [Discovering Options for Exploration by Minimizing Cover Time](https://mlsanity.com/paper/1903.00606/) (2019)
- [Data-efficient Hindsight Off-policy Option Learning](https://mlsanity.com/paper/2007.15588/) (2020)
- [Options of Interest: Temporal Abstraction with Interest Functions](https://mlsanity.com/paper/2001.00271/) (2020)
- [Unsupervised Skill Discovery with Bottleneck Option Learning](https://mlsanity.com/paper/2106.14305/) (2021)
- [Flexible Option Learning](https://mlsanity.com/paper/2112.03097/) (2021)
- [Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing](https://mlsanity.com/paper/2107.05545/) (2021)

## Built on by (most cited first, 29 of 29)

- [Motif: Intrinsic Motivation from Artificial Intelligence Feedback](https://mlsanity.com/paper/2310.00166/) (2023)
- [Foundation Policies with Hilbert Representations](https://mlsanity.com/paper/2402.15567/) (2024)
- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [MaestroMotif: Skill Design from Artificial Intelligence Feedback](https://mlsanity.com/paper/2412.08542/) (2024)
- [Episodic Novelty Through Temporal Distance](https://mlsanity.com/paper/2501.15418/) (2025)
- [Diversifying AI: Towards Creative Chess with AlphaZero](https://mlsanity.com/paper/2308.09175/) (2023)
- [Rethinking Decision Transformer via Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2311.00267/) (2023)
- [Proper Laplacian Representation Learning](https://mlsanity.com/paper/2310.10833/) (2023)
- [CQM: Curriculum Reinforcement Learning with a Quantized World Model](https://mlsanity.com/paper/2310.17330/) (2023)
- [SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions](https://mlsanity.com/paper/2410.18416/) (2024)
- [Beyond Optimism: Exploration With Partially Observable Rewards](https://mlsanity.com/paper/2406.13909/) (2024)
- [ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning](https://mlsanity.com/paper/2510.14176/) (2025)
- [Laplacian Representations for Decision-Time Planning](https://mlsanity.com/paper/2602.05031/) (2026)
- [Reward-Aware Proto-Representations in Reinforcement Learning](https://mlsanity.com/paper/2505.16217/) (2025)
- [Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representations](https://mlsanity.com/paper/2511.10872/) (2025)
- [Memory, Space, and Planning: Multiscale Predictive Representations](https://mlsanity.com/paper/2401.09491/) (2024)
- [Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models](https://mlsanity.com/paper/2505.07815/) (2025)
- [InnateCoder: Learning Programmatic Options with Foundation Models](https://mlsanity.com/paper/2505.12508/) (2025)
- [Iterative Option Discovery for Planning, by Planning](https://mlsanity.com/paper/2310.01569/) (2023)
- [On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/2404.07826/) (2024)
- [Open Problem: Active Representation Learning](https://mlsanity.com/paper/2406.03845/) (2024)
- [The Laplacian Keyboard: Beyond the Linear Span](https://mlsanity.com/paper/2602.07730/) (2026)
- [Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery](https://mlsanity.com/paper/2609.36473/) (2026)
- [Unveiling Options with Neural Decomposition](https://mlsanity.com/paper/2410.11262/) (2024)
- [Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines](https://mlsanity.com/paper/2510.27329/) (2025)
- [DROGO: Default Representation Objective via Graph Optimization in Reinforcement Learning](https://mlsanity.com/paper/2602.00403/) (2026)
- [From Connectivity to Rewards: Dense Reward Learning with Directed State Graphs](https://mlsanity.com/paper/2609.10781/) (2026)
- [Mastering Atari 2600 Games with Discovered Options](https://mlsanity.com/paper/2610.03604/) (2026)

## Abstract (arXiv)

Selecting exploratory actions that generate a rich stream of experience for better learning is a fundamental challenge in reinforcement learning (RL). An approach to tackle this problem consists in selecting actions according to specific policies for an extended period of time, also known as options. A recent line of work to derive such exploratory options builds upon the eigenfunctions of the graph Laplacian. Importantly, until now these methods have been mostly limited to tabular domains where (1) the graph Laplacian matrix was either given or could be fully estimated, (2) performing eigendecomposition on this matrix was computationally tractable, and (3) value functions could be learned exactly. Additionally, these methods required a separate option discovery phase. These assumptions are fundamentally not scalable. In this paper we address these limitations and show how recent results for directly approximating the eigenfunctions of the Laplacian can be leveraged to truly scale up options-based exploration. To do so, we introduce a fully online deep RL algorithm for discovering Laplacian-based options and evaluate our approach on a variety of pixel-based tasks. We compare to several state-of-the-art exploration methods and show that our approach is effective, general, and especially promising in non-stationary settings.