mlsanityData to 7 Oct 2026

Geometric Entropic Exploration

arXiv 2101.02055 ↗First posted 6 Jan 2021cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

Exploration is essential for solving complex Reinforcement Learning (RL) tasks. Maximum State-Visitation Entropy (MSVE) formulates the exploration problem as a well-defined policy optimization problem whose solution aims at visiting all states as uniformly as possible. This is in contrast to standard uncertainty-based approaches where exploration is transient and eventually vanishes. However, existing approaches to MSVE are theoretically justified only for discrete state-spaces as they are oblivious to the geometry of continuous domains. We address this challenge by introducing Geometric Entropy Maximisation (GEM), a new algorithm that maximises the geometry-aware Shannon entropy of state-visits in both discrete and continuous domains. Our key theoretical contribution is casting geometry-aware MSVE exploration as a tractable problem of optimising a simple and novel noise-contrastive objective function. In our experiments, we show the efficiency of GEM in solving several RL problems with sparse rewards, compared against other deep RL exploration approaches.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 32 of its 58 references that are in the map. First posted on arXiv 6 Jan 2021.

Builds on

32 of 58 references.

References · 32

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  8. 2016Deep Exploration via Bootstrapped DQNcited by 518
  9. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  10. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  11. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 284
  12. 2016Variational Intrinsic Controlcited by 262
20 more
  1. 2017Count-Based Exploration with Neural Density Modelscited by 255
  2. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  3. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  4. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  5. 2019Efficient Exploration via State Marginal Matchingcited by 186
  6. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  7. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  8. 2017Optimizing the Latent Space of Generative Networkscited by 157
  9. 2019Behaviour Suite for Reinforcement Learningcited by 129
  10. 2018State Representation Learning for Control: An Overviewcited by 125
  11. 2018Episodic Curiosity through Reachabilitycited by 109
  12. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
  13. 2019Wasserstein Dependency Measure for Representation Learningcited by 69
  14. 2020Near-optimal Regret Bounds for Stochastic Shortest Pathcited by 50
  15. 2019No-Regret Exploration in Goal-Oriented Reinforcement Learningcited by 34
  16. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  17. 2019Marginalized State Distribution Entropy Regularization in Policy Optimizationcited by 16
  18. 2019Geometric Losses for Distributional Learningcited by 7
  19. 2019GAIT: A Geometric Approach to Information Theorycited by 5
  20. 2019Autonomous exploration for navigating in non-stationary CMPscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 11; 2023: 6; 2024: 4; 2025: 4; 2026: 3.

32 papers.

Sort:

By area:

  1. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  2. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  3. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
  4. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
  5. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  6. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  7. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  8. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  9. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  10. 2022Time to augment self-supervised visual representation learningcited by 15
  11. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 12
  12. 2022k-Means Maximum Entropy Explorationcited by 11
20 more
  1. 2021Action Redundancy in Reinforcement Learningcited by 9
  2. 2023Unlocking the Power of Representations in Long-term Novelty-based Explorationcited by 8
  3. 2021DisTop: Discovering a Topological representation to learn diverse and rewarding skillscited by 7
  4. 2024How to Explore with Belief: State Entropy Maximization in POMDPscited by 6
  5. 2024The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enoughcited by 6
  6. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
  7. 2026Verifier-Constrained Flow Expansion for Discovery Beyond the Datacited by 6
  8. 2022Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environmentscited by 4
  9. 2023Policy Gradient Algorithms Implicitly Optimize by Continuationcited by 4
  10. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learningcited by 4
  11. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Storycited by 4
  12. 2022POLTER: Policy Trajectory Ensemble Regularization for Unsupervised Reinforcement Learningcited by 3
  13. 2024Behind the Myth of Exploration in Policy Gradientscited by 3
  14. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measurescited by 3
  15. 2023Influence of the Geometry of the world model on Curiosity Based Explorationcited by 1
  16. 2021More Efficient Exploration with Symbolic Priors on Action Sequence Equivalencescited by 0
  17. 2022On the importance of data collection for training general goal-reaching policiescited by 0
  18. 2022Quality-diversity in dissimilarity spacescited by 0
  19. 2026K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agentscited by 0
  20. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measurescited by 0