mlsanityData to 14 Oct 2026

Provably Efficient Maximum Entropy Exploration

arXiv 1812.02690 ↗First posted 6 Dec 2018cs.LG cs.AI stat.MLcited by 242

Not on the map. Why?

Cited by 68 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Suppose an agent is in a (possibly unknown) Markov Decision Process in the absence of a reward signal, what might we hope that an agent can efficiently learn to do? This work studies a broad class of objectives that are defined solely as functions of the state-visitation frequencies that are induced by how the agent behaves. For example, one natural, intrinsically defined, objective problem is for the agent to learn a policy which induces a distribution over state space that is as uniform as possible, which can be measured in an entropic sense. We provide an efficient algorithm to optimize such such intrinsically defined objectives, when given access to a black box planning oracle (which is robust to function approximation). Furthermore, when restricted to the tabular setting where we have sample based access to the MDP, our proposed algorithm is provably efficient, both in terms of its sample and computational complexities. Key to our algorithmic methodology is utilizing the conditional gradient method (a.k.a. the Frank-Wolfe algorithm) which utilizes an approximate MDP solver.

Key facts. Cited by 242 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 44; 2022: 38; 2023: 49; 2024: 42; 2025: 34; 2026: 34.

242 papers.

Sort:

By area:

  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  2. 2021Reinforcement Learning with Prototypical Representationscited by 179
  3. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  4. 2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 132
  5. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 121
  6. 2021APS: Active Pretraining with Successor Featurescited by 112
  7. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  8. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 108
  9. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  10. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  11. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 88
  12. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
38 more
  1. 2021Reward is enough for convex MDPscited by 84
  2. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 82
  3. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 73
  4. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  5. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 70
  6. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  7. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  8. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
  9. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  10. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 47
  11. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  12. 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
  13. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  14. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  15. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 37
  16. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 37
  17. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  18. 2022Instance-Dependent Near-Optimal Policy Identification in Linear MDPs via Online Experiment Designcited by 33
  19. 2023Fast Rates for Maximum Entropy Explorationcited by 33
  20. 2021Geometric Entropic Explorationcited by 32
  21. 2021Regularized Policies are Reward Robustcited by 32
  22. 2023Thompson sampling for improved exploration in GFlowNetscited by 32
  23. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  24. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 30
  25. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  26. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 28
  27. 2023Distributional GFlowNets with Quantile Flowscited by 28
  28. 2024Autoregressive Action Sequence Learning for Robotic Manipulationcited by 27
  29. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
  30. 2022Any-Play: An Intrinsic Augmentation for Zero-Shot Coordinationcited by 23
  31. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  32. 2023Submodular Reinforcement Learningcited by 23
  33. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimizationcited by 22
  34. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 22
  35. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  36. 2021Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learningcited by 21
  37. 2022A Differential Entropy Estimator for Training Neural Networkscited by 20
  38. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20

50 most cited of 242 All 242 (JSON)