# Geometric Entropic Exploration

arXiv: https://arxiv.org/abs/2101.02055 · HTML: https://mlsanity.com/paper/2101.02055/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 58 references that are in the map.
- First posted on arXiv 6 Jan 2021.

## Builds on (most cited first, 30 of 32)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Minimax Regret Bounds for Reinforcement Learning](https://mlsanity.com/paper/1703.05449/) (2017)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Count-Based Exploration with Neural Density Models](https://mlsanity.com/paper/1703.01310/) (2017)
- [Provably Efficient Maximum Entropy Exploration](https://mlsanity.com/paper/1812.02690/) (2018)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Efficient Exploration via State Marginal Matching](https://mlsanity.com/paper/1906.05274/) (2019)
- [Reward-Free Exploration for Reinforcement Learning](https://mlsanity.com/paper/2002.02794/) (2020)
- [Skew-Fit: State-Covering Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/1903.03698/) (2019)
- [Optimizing the Latent Space of Generative Networks](https://mlsanity.com/paper/1707.05776/) (2017)
- [Behaviour Suite for Reinforcement Learning](https://mlsanity.com/paper/1908.03568/) (2019)
- [State Representation Learning for Control: An Overview](https://mlsanity.com/paper/1802.04181/) (2018)
- [Episodic Curiosity through Reachability](https://mlsanity.com/paper/1810.02274/) (2018)
- [Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning](https://mlsanity.com/paper/1510.08906/) (2015)
- [Wasserstein Dependency Measure for Representation Learning](https://mlsanity.com/paper/1903.11780/) (2019)
- [Near-optimal Regret Bounds for Stochastic Shortest Path](https://mlsanity.com/paper/2002.09869/) (2020)
- [No-Regret Exploration in Goal-Oriented Reinforcement Learning](https://mlsanity.com/paper/1912.03517/) (2019)
- [An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policies](https://mlsanity.com/paper/1907.04662/) (2019)
- [Marginalized State Distribution Entropy Regularization in Policy Optimization](https://mlsanity.com/paper/1912.05128/) (2019)
- [Geometric Losses for Distributional Learning](https://mlsanity.com/paper/1905.06005/) (2019)

## Built on by (most cited first, 30 of 32)

- [BYOL-Explore: Exploration by Bootstrapped Prediction](https://mlsanity.com/paper/2206.08332/) (2022)
- [The Importance of Non-Markovianity in Maximum State Entropy Exploration](https://mlsanity.com/paper/2202.03060/) (2022)
- [Challenging Common Assumptions in Convex Reinforcement Learning](https://mlsanity.com/paper/2202.01511/) (2022)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Maximum State Entropy Exploration using Predecessor and Successor Representations](https://mlsanity.com/paper/2306.14808/) (2023)
- [Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration](https://mlsanity.com/paper/2305.19476/) (2023)
- [Pretraining in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2211.03959/) (2022)
- [Unsupervised Reinforcement Learning in Multiple Environments](https://mlsanity.com/paper/2112.08746/) (2021)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Time to augment self-supervised visual representation learning](https://mlsanity.com/paper/2207.13492/) (2022)
- [Provable Maximum Entropy Manifold Exploration via Diffusion Models](https://mlsanity.com/paper/2506.15385/) (2025)
- [k-Means Maximum Entropy Exploration](https://mlsanity.com/paper/2205.15623/) (2022)
- [Action Redundancy in Reinforcement Learning](https://mlsanity.com/paper/2102.11329/) (2021)
- [Unlocking the Power of Representations in Long-term Novelty-based Exploration](https://mlsanity.com/paper/2305.01521/) (2023)
- [DisTop: Discovering a Topological representation to learn diverse and rewarding skills](https://mlsanity.com/paper/2106.03853/) (2021)
- [Verifier-Constrained Flow Expansion for Discovery Beyond the Data](https://mlsanity.com/paper/2602.15984/) (2026)
- [How to Explore with Belief: State Entropy Maximization in POMDPs](https://mlsanity.com/paper/2406.02295/) (2024)
- [The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enough](https://mlsanity.com/paper/2406.12795/) (2024)
- [State Entropy Regularization for Robust Reinforcement Learning](https://mlsanity.com/paper/2506.07085/) (2025)
- [Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments](https://mlsanity.com/paper/2211.10515/) (2022)
- [Policy Gradient Algorithms Implicitly Optimize by Continuation](https://mlsanity.com/paper/2305.06851/) (2023)
- [Towards Principled Unsupervised Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2502.08365/) (2025)
- [Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Story](https://mlsanity.com/paper/2505.01336/) (2025)
- [POLTER: Policy Trajectory Ensemble Regularization for Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2205.11357/) (2022)
- [Behind the Myth of Exploration in Policy Gradients](https://mlsanity.com/paper/2402.00162/) (2024)
- [Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures](https://mlsanity.com/paper/2412.06655/) (2024)
- [Influence of the Geometry of the world model on Curiosity Based Exploration](https://mlsanity.com/paper/2304.00188/) (2023)
- [More Efficient Exploration with Symbolic Priors on Action Sequence Equivalences](https://mlsanity.com/paper/2110.10632/) (2021)
- [On the importance of data collection for training general goal-reaching policies](https://mlsanity.com/paper/2211.03521/) (2022)
- [Quality-diversity in dissimilarity spaces](https://mlsanity.com/paper/2211.12337/) (2022)

## Abstract (arXiv)

Exploration is essential for solving complex Reinforcement Learning (RL) tasks. Maximum State-Visitation Entropy (MSVE) formulates the exploration problem as a well-defined policy optimization problem whose solution aims at visiting all states as uniformly as possible. This is in contrast to standard uncertainty-based approaches where exploration is transient and eventually vanishes. However, existing approaches to MSVE are theoretically justified only for discrete state-spaces as they are oblivious to the geometry of continuous domains. We address this challenge by introducing Geometric Entropy Maximisation (GEM), a new algorithm that maximises the geometry-aware Shannon entropy of state-visits in both discrete and continuous domains. Our key theoretical contribution is casting geometry-aware MSVE exploration as a tractable problem of optimising a simple and novel noise-contrastive objective function. In our experiments, we show the efficiency of GEM in solving several RL problems with sparse rewards, compared against other deep RL exploration approaches.