mlsanityData to 7 Oct 2026

State Entropy Maximization with Random Encoders for Efficient Exploration

arXiv 2102.09430 ↗First posted 18 Feb 2021cs.LGcited by 107

Not on the map. Why?

Abstract (arXiv)

Recent exploration methods have proven to be a recipe for improving sample-efficiency in deep reinforcement learning (RL). However, efficient exploration in high-dimensional observation spaces still remains a challenge. This paper presents Random Encoders for Efficient Exploration (RE3), an exploration method that utilizes state entropy as an intrinsic reward. In order to estimate state entropy in environments with high-dimensional observations, we utilize a k-nearest neighbor entropy estimator in the low-dimensional representation space of a convolutional encoder. In particular, we find that the state entropy can be estimated in a stable and compute-efficient manner by utilizing a randomly initialized encoder, which is fixed throughout training. Our experiments show that RE3 significantly improves the sample-efficiency of both model-free and model-based RL methods on locomotion and navigation tasks from DeepMind Control Suite and MiniGrid benchmarks. We also show that RE3 allows learning diverse behaviors without extrinsic rewards, effectively improving sample-efficiency in downstream tasks. Source code and videos are available at https://sites.google.com/view/re3-rl.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 18 Feb 2021.

Builds on

43 of 59 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  9. 2017Deep Image Priorcited by 1,097
  10. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
31 more
  1. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  5. 2020Reinforcement Learning with Augmented Datacited by 452
  6. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  7. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  8. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  9. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  10. 2019Self-Supervised Exploration via Disagreementcited by 263
  11. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  12. 2017Noisy Networks for Explorationcited by 262
  13. 2017Count-Based Exploration with Neural Density Modelscited by 255
  14. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  15. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  16. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  17. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  18. 2019Efficient Exploration via State Marginal Matchingcited by 186
  19. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  20. 2017Parameter Space Noise for Explorationcited by 158
  21. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  22. 2017CondenseNet: An Efficient DenseNet using Learned Group Convolutionscited by 121
  23. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
  24. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 78
  25. 2019Weight Agnostic Neural Networkscited by 71
  26. 2019No Training Required: Exploring Random Encoders for Sentence Classificationcited by 30
  27. 2020Novelty Search in Representational Space for Sample Efficient Explorationcited by 26
  28. 2020Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimatecited by 21
  29. 2019Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environmentcited by 17
  30. 2020Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensemblescited by 14
  31. 2019Training CNNs with Selective Allocation of Channelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 19; 2023: 25; 2024: 17; 2025: 16; 2026: 10.

107 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  4. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  5. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  6. 2021APS: Active Pretraining with Successor Featurescited by 109
  7. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  8. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  9. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  10. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  11. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  12. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
38 more
  1. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
  2. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  3. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  4. 2021Learning more skills through optimistic explorationcited by 37
  5. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 36
  6. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  7. 2023Bridging RL Theory and Practice with the Effective Horizoncited by 31
  8. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  9. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
  10. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
  11. 2021Curiosity-Driven Exploration via Latent Bayesian Surprisecited by 22
  12. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  13. 2021Dynamic Bottleneck for Robust Self-Supervised Explorationcited by 21
  14. 2023Explore to Generalize in Zero-Shot RLcited by 20
  15. 2022R'enyi State Entropy for Exploration Acceleration in Reinforcement Learningcited by 18
  16. 2022Unsupervised Representation Learning in Deep Reinforcement Learning: A Reviewcited by 18
  17. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  18. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  19. 2023FOCUS: Object-Centric World Models for Robotics Manipulationcited by 17
  20. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  21. 2021Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillationcited by 15
  22. 2025Episodic Novelty Through Temporal Distancecited by 15
  23. 2023COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RLcited by 14
  24. 2022Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualizationcited by 13
  25. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  26. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 12
  27. 2022k-Means Maximum Entropy Explorationcited by 11
  28. 2023Automatic Intrinsic Reward Shaping for Exploration in Deep Reinforcement Learningcited by 11
  29. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  30. 2024RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learningcited by 10
  31. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  32. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Explorationcited by 9
  33. 2021Learning Altruistic Behaviours in Reinforcement Learning without External Rewardscited by 8
  34. 2022EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Modelcited by 8
  35. 2023Unlocking the Power of Representations in Long-term Novelty-based Explorationcited by 8
  36. 2023Maximum diffusion reinforcement learningcited by 8
  37. 2023Variational Curriculum Reinforcement Learning for Unsupervised Discovery of Skillscited by 8
  38. 2023Learning to Discover Skills through Guidancecited by 8

50 most cited of 107 All 107 (JSON)