mlsanityData to 13 Oct 2026

Deep Laplacian-based Options for Temporally-Extended Exploration

arXiv 2301.11181 ↗First posted 26 Jan 2023cs.LG cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

Selecting exploratory actions that generate a rich stream of experience for better learning is a fundamental challenge in reinforcement learning (RL). An approach to tackle this problem consists in selecting actions according to specific policies for an extended period of time, also known as options. A recent line of work to derive such exploratory options builds upon the eigenfunctions of the graph Laplacian. Importantly, until now these methods have been mostly limited to tabular domains where (1) the graph Laplacian matrix was either given or could be fully estimated, (2) performing eigendecomposition on this matrix was computationally tractable, and (3) value functions could be learned exactly. Additionally, these methods required a separate option discovery phase. These assumptions are fundamentally not scalable. In this paper we address these limitations and show how recent results for directly approximating the eigenfunctions of the Laplacian can be leveraged to truly scale up options-based exploration. To do so, we introduce a fully online deep RL algorithm for discovering Laplacian-based options and evaluate our approach on a variety of pixel-based tasks. We compare to several state-of-the-art exploration methods and show that our approach is effective, general, and especially promising in non-stationary settings.

Key facts. Cited by 29 papers in the map (data through 13 Oct 2026). Builds on 35 of its 65 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

35 of 65 references.

References · 35

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,221
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,710
  3. 2018Exploration by Random Network Distillationcited by 971
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 777
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 739
  6. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  7. 2016The Option-Critic Architecturecited by 439
  8. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 332
  9. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 271
  10. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 268
  11. 2016Variational Intrinsic Controlcited by 264
  12. 2020First return, then explorecited by 257
23 more
  1. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  2. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 109
  3. 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
  4. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 85
  5. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 81
  6. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  7. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 61
  8. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 58
  9. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  10. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 53
  11. 2018Spectral Inference Networks: Unifying Deep and Spectral Learningcited by 35
  12. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  13. 2019Discovering Options for Exploration by Minimizing Cover Timecited by 31
  14. 2020Data-efficient Hindsight Off-policy Option Learningcited by 31
  15. 2020Options of Interest: Temporal Abstraction with Interest Functionscited by 30
  16. 2021Unsupervised Skill Discovery with Bottleneck Option Learningcited by 28
  17. 2021Flexible Option Learningcited by 25
  18. 2021Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawingcited by 23
  19. 2022Neural Eigenfunctions Are Structured Representation Learnerscited by 16
  20. 2020Reward Propagation Using Graph Convolutional Networkscited by 15
  21. 2022Temporal Abstractions-Augmented Temporally Contrastive Learning: An Alternative to the Laplacian in RLcited by 7
  22. 2016Learning Purposeful Behaviour in the Absence of Rewardscited by 6
  23. 2020Option Discovery in the Absence of Rewards with Manifold Analysiscited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 7; 2024: 8; 2025: 8; 2026: 6.

29 papers.

Sort:

By area:

  1. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 101
  2. 2024Foundation Policies with Hilbert Representationscited by 92
  3. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 82
  4. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 52
  5. 2024MaestroMotif: Skill Design from Artificial Intelligence Feedbackcited by 24
  6. 2025Episodic Novelty Through Temporal Distancecited by 16
  7. 2023Diversifying AI: Towards Creative Chess with AlphaZerocited by 15
  8. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
  9. 2023CQM: Curriculum Reinforcement Learning with a Quantized World Modelcited by 11
  10. 2023Proper Laplacian Representation Learningcited by 10
  11. 2024SkiLD: Unsupervised Skill Discovery Guided by Factor Interactionscited by 9
  12. 2024Beyond Optimism: Exploration With Partially Observable Rewardscited by 6
17 more
  1. 2025ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learningcited by 5
  2. 2026Laplacian Representations for Decision-Time Planningcited by 5
  3. 2025Reward-Aware Proto-Representations in Reinforcement Learningcited by 4
  4. 2025Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representationscited by 3
  5. 2024Memory, Space, and Planning: Multiscale Predictive Representationscited by 2
  6. 2025Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Modelscited by 2
  7. 2025InnateCoder: Learning Programmatic Options with Foundation Modelscited by 2
  8. 2023Iterative Option Discovery for Planning, by Planningcited by 1
  9. 2024On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learningcited by 1
  10. 2024Open Problem: Active Representation Learningcited by 1
  11. 2026The Laplacian Keyboard: Beyond the Linear Spancited by 1
  12. 2026Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discoverycited by 1
  13. 2024Unveiling Options with Neural Decompositioncited by 0
  14. 2025Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machinescited by 0
  15. 2026DROGO: Default Representation Objective via Graph Optimization in Reinforcement Learningcited by 0
  16. 2026From Connectivity to Rewards: Dense Reward Learning with Directed State Graphscited by 0
  17. 2026Mastering Atari 2600 Games with Discovered Optionscited by 0