mlsanityData to 13 Oct 2026

Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning

arXiv 1810.06530 ↗First posted 15 Oct 2018cs.LG stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

Posterior sampling for reinforcement learning (PSRL) is an effective method for balancing exploration and exploitation in reinforcement learning. Randomised value functions (RVF) can be viewed as a promising approach to scaling PSRL. However, we show that most contemporary algorithms combining RVF with neural network function approximation do not possess the properties which make PSRL effective, and provably fail in sparse reward problems. Moreover, we find that propagation of uncertainty, a property of PSRL previously thought important for exploration, does not preclude this failure. We use these insights to design Successor Uncertainties (SU), a cheap and easy to implement RVF algorithm that retains key properties of PSRL. SU is highly effective on hard tabular exploration benchmarks. Furthermore, on the Atari 2600 domain, it surpasses human performance on 38 of 49 games tested (achieving a median human normalised score of 2.09), and outperforms its closest RVF competitor, Bootstrapped DQN, on 36 of those.

Key facts. Cited by 35 papers in the map (data through 13 Oct 2026). First posted on arXiv 15 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 8; 2022: 8; 2023: 8; 2024: 4; 2025: 5; 2026: 2.

35 papers.

Sort:

By area:

  1. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  2. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  3. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 102
  4. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 69
  5. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  6. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  7. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  8. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23
  9. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  10. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  11. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
  12. 2021Active Hierarchical Exploration with Stable Subgoal Representation Learningcited by 17
23 more
  1. 2023Posterior Sampling for Deep Reinforcement Learningcited by 16
  2. 2021Risk-Aware Transfer in Reinforcement Learning using Successor Featurescited by 12
  3. 2023Combining Behaviors with the Successor Features Keyboardcited by 12
  4. 2023Diverse Projection Ensembles for Distributional Reinforcement Learningcited by 8
  5. 2022Epistemic Monte Carlo Tree Searchcited by 7
  6. 2023Composing Task Knowledge with Modular Successor Feature Approximatorscited by 7
  7. 2024World Models with Hints of Large Language Models for Goal Achievingcited by 6
  8. 2023Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learningcited by 5
  9. 2025The Sample Complexity of Online Reinforcement Learning: A Multi-model Perspectivecited by 5
  10. 2022Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experiencecited by 4
  11. 2023Information-theoretic Analysis of Test Data Sensitivity in Uncertaintycited by 3
  12. 2023Bayesian Exploration Networkscited by 3
  13. 2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3
  14. 2024A Bayesian Solution To The Imitation Gapcited by 2
  15. 2023Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learningcited by 1
  16. 2025Universal Value-Function Uncertaintiescited by 1
  17. 2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1
  18. 2022A Generalized Bootstrap Target for Value-Learning, Efficiently Combining Value and Feature Predictionscited by 0
  19. 2022Excess risk analysis for epistemic uncertainty with application to variational inferencecited by 0
  20. 2024Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learningcited by 0
  21. 2025Concurrent Learning with Aggregated States via Randomized Least Squares Value Iterationcited by 0
  22. 2025Towards Fully Automated Decision-Making Systems for Greenhouse Control: Challenges and Opportunitiescited by 0
  23. 2026Deep Epistemic Value Functions for Optimistic Explorationcited by 0