mlsanityData to 7 Oct 2026

Count-Based Exploration with the Successor Representation

arXiv 1807.11622 ↗First posted 31 Jul 2018cs.LG cs.AI stat.MLcited by 112

Not on the map. Why?

Abstract (arXiv)

In this paper we introduce a simple approach for exploration in reinforcement learning (RL) that allows us to develop theoretically justified algorithms in the tabular case but that is also extendable to settings where function approximation is required. Our approach is based on the successor representation (SR), which was originally introduced as a representation defining state generalization by the similarity of successor states. Here we show that the norm of the SR, while it is being learned, can be used as a reward bonus to incentivize exploration. In order to better understand this transient behavior of the norm of the SR we introduce the substochastic successor representation (SSR) and we show that it implicitly counts the number of times each state (or feature) has been observed. We use this result to introduce an algorithm that performs as well as some theoretically sample-efficient approaches. Finally, we extend these ideas to a deep RL algorithm and show that it achieves state-of-the-art performance in Atari 2600 games when in a low sample-complexity regime.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 29; 2023: 23; 2024: 16; 2025: 15; 2026: 8.

112 papers.

Sort:

By area:

  1. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  2. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  3. 2021APS: Active Pretraining with Successor Featurescited by 109
  4. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
  5. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  6. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 52
  7. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  8. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  9. 2022Exploration via Elliptical Episodic Bonusescited by 51
  10. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  11. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  12. 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
38 more
  1. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  2. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  3. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  4. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  5. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
  6. 2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
  7. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23
  8. 2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 23
  9. 2021Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learningcited by 23
  10. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  11. 2021Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawingcited by 22
  12. 2022Multi-Stage Episodic Control for Strategic Exploration in Text Gamescited by 22
  13. 2021A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learningcited by 21
  14. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
  15. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
  16. 2025Temporal Difference Flowscited by 20
  17. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  18. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 19
  19. 2022Unsupervised Representation Learning in Deep Reinforcement Learning: A Reviewcited by 18
  20. 2021Active Hierarchical Exploration with Stable Subgoal Representation Learningcited by 17
  21. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
  22. 2021Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Drivingcited by 16
  23. 2023Curious Replay for Model-based Adaptationcited by 16
  24. 2023Model-based Offline Reinforcement Learning with Count-based Conservatismcited by 16
  25. 2021Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learningcited by 15
  26. 2024A Distributional Analogue to the Successor Representationcited by 15
  27. 2025Episodic Novelty Through Temporal Distancecited by 15
  28. 2021Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learningcited by 14
  29. 2021A First-Occupancy Representation for Reinforcement Learningcited by 14
  30. 2022Towards Understanding How Machines Can Learn Causal Overhypothesescited by 13
  31. 2022LECO: Learnable Episodic Count for Task-Specific Intrinsic Rewardcited by 11
  32. 2023Automatic Intrinsic Reward Shaping for Exploration in Deep Reinforcement Learningcited by 11
  33. 2022Reachability-Aware Laplacian Representation in Reinforcement Learningcited by 10
  34. 2024RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learningcited by 10
  35. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  36. 2022Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learningcited by 9
  37. 2022Foundation Models for Semantic Novelty in Reinforcement Learningcited by 9
  38. 2023Sample Efficient Reinforcement Learning in Mixed Systems through Augmented Samples and Its Applications to Queueing Networkscited by 9

50 most cited of 112 All 112 (JSON)