mlsanityData to 14 Oct 2026

Maximum State Entropy Exploration using Predecessor and Successor Representations

arXiv 2306.14808 ↗First posted 26 Jun 2023cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Animals have a developed ability to explore that aids them in important tasks such as locating food, exploring for shelter, and finding misplaced items. These exploration skills necessarily track where they have been so that they can plan for finding items with relative efficiency. Contemporary exploration algorithms often learn a less efficient exploration strategy because they either condition only on the current state or simply rely on making random open-loop exploratory moves. In this work, we propose ηpsi-Learning, a method to learn efficient exploratory policies by conditioning on past episodic experience to make the next exploratory move. Specifically, ηpsi-Learning learns an exploration policy that maximizes the entropy of the state visitation distribution of a single trajectory. Furthermore, we demonstrate how variants of the predecessor representation and successor representations can be combined to predict the state visitation entropy. Our experiments demonstrate the efficacy of ηpsi-Learning to strategically explore the environment and maximize the state coverage with limited samples.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 50 of its 72 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

50 of 72 references.

References · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2023Mastering Diverse Domains through World Modelscited by 1,253
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  9. 2020Mastering Atari with Discrete World Modelscited by 1,007
  10. 2018Exploration by Random Network Distillationcited by 993
  11. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  12. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
38 more
  1. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  2. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  3. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  4. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  5. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  6. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  7. 2020First return, then explorecited by 266
  8. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  9. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  10. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  11. 2019Efficient Exploration via State Marginal Matchingcited by 188
  12. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  13. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  14. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  15. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  16. 2021Learning One Representation to Optimize All Rewardscited by 115
  17. 2018Count-Based Exploration with the Successor Representationcited by 112
  18. 2018Universal Successor Features Approximatorscited by 103
  19. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  20. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  21. 2020Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrificescited by 59
  22. 2016Deep Reinforcement Learning with Successor Features for Navigation across Similar Environmentscited by 50
  23. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 47
  24. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 42
  25. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  26. 2021Geometric Entropic Explorationcited by 32
  27. 2019Active Exploration in Markov Decision Processescited by 27
  28. 2020Expected Eligibility Tracescited by 27
  29. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  30. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 20
  31. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  32. 2021A First-Occupancy Representation for Reinforcement Learningcited by 14
  33. 2021A New Representation of Successor Features for Transfer across Dissimilar Environmentscited by 13
  34. 2017Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learningcited by 10
  35. 2018Universal Successor Representations for Transfer Reinforcement Learningcited by 10
  36. 2022Learning Robust Dynamics through Variational Sparse Gatingcited by 8
  37. 2019VUSFA:Variational Universal Successor Features Approximator to Improve Transfer DRL for Target Driven Visual Navigationcited by 4
  38. 2022Predecessor Featurescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 9; 2026: 6.

23 papers.

Sort:

By area:

  1. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 26
  2. 2025Temporal Difference Flowscited by 24
  3. 2024A Distributional Analogue to the Successor Representationcited by 17
  4. 2025Horizon Generalization in Reinforcement Learningcited by 17
  5. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  6. 2025Intention-Conditioned Flow Occupancy Modelscited by 9
  7. 2023Improving Intrinsic Exploration by Creating Stationary Objectivescited by 7
  8. 2024How to Explore with Belief: State Entropy Maximization in POMDPscited by 6
  9. 2024The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enoughcited by 6
  10. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
  11. 2024Surprise-Adaptive Intrinsic Motivation for Unsupervised Reinforcement Learningcited by 4
  12. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learningcited by 4
11 more
  1. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Storycited by 4
  2. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measurescited by 3
  3. 2026Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilitiescited by 3
  4. 2025AMPED: Adaptive Multi-objective Projection for balancing Exploration and skill Diversificationcited by 1
  5. 2025SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimationcited by 1
  6. 2026Maximum Entropy Exploration Without the Rolloutscited by 1
  7. 2025Accelerated Online Reinforcement Learning using Auxiliary Start State Distributionscited by 0
  8. 2026K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agentscited by 0
  9. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measurescited by 0
  10. 2026Global Optimality for Constrained Exploration via Penalty Regularizationcited by 0
  11. 2026SUN: Reaching for Novelty in Reinforcement Learningcited by 0