mlsanityData to 7 Oct 2026

Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration

arXiv 2305.19476 ↗First posted 31 May 2023cs.LG cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

A promising technique for exploration is to maximize the entropy of visited state distribution, i.e., state entropy, by encouraging uniform coverage of visited state space. While it has been effective for an unsupervised setup, it tends to struggle in a supervised setup with a task reward, where an agent prefers to visit high-value states to exploit the task reward. Such a preference can cause an imbalance between the distributions of high-value states and low-value states, which biases exploration towards low-value state regions as a result of the state entropy increasing when the distribution becomes more uniform. This issue is exacerbated when high-value states are narrowly distributed within the state space, making it difficult for the agent to complete the tasks. In this paper, we present a novel exploration technique that maximizes the value-conditional state entropy, which separately estimates the state entropies that are conditioned on the value estimates of each state, then maximizes their average. By only considering the visited states with similar value estimates for computing the intrinsic bonus, our method prevents the distribution of low-value states from affecting exploration around high-value states, and vice versa. We demonstrate that the proposed alternative to the state entropy baseline significantly accelerates various reinforcement learning algorithms across a variety of tasks within MiniGrid, DeepMind Control Suite, and Meta-World benchmarks. Source code is available at https://sites.google.com/view/rl-vcse.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

43 of 63 references.

References · 43

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2023Mastering Diverse Domains through World Modelscited by 1,193
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2020Mastering Atari with Discrete World Modelscited by 990
  8. 2018Exploration by Random Network Distillationcited by 966
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  12. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
31 more
  1. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  2. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  3. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  4. 2019Self-Supervised Exploration via Disagreementcited by 263
  5. 2017Noisy Networks for Explorationcited by 262
  6. 2017Count-Based Exploration with Neural Density Modelscited by 255
  7. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  8. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  9. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  10. 2019Efficient Exploration via State Marginal Matchingcited by 186
  11. 2021Reinforcement Learning with Prototypical Representationscited by 179
  12. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  13. 2022Masked World Models for Visual Controlcited by 169
  14. 2017Parameter Space Noise for Explorationcited by 158
  15. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  16. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  17. 2021APS: Active Pretraining with Successor Featurescited by 109
  18. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  19. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  20. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  21. 2021Geometric Entropic Explorationcited by 32
  22. 2020Novelty Search in Representational Space for Sample Efficient Explorationcited by 26
  23. 2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
  24. 2020Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimatecited by 21
  25. 2023Deep RL at Scale: Sorting Waste in Office Buildings with a Fleet of Mobile Manipulatorscited by 21
  26. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  27. 2022R'enyi State Entropy for Exploration Acceleration in Reinforcement Learningcited by 18
  28. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  29. 2020Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensemblescited by 14
  30. 2021Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learningcited by 14
  31. 2022k-Means Maximum Entropy Explorationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 10; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  2. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  3. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  4. 2025EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Controlcited by 13
  5. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  6. 2025Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Drivingcited by 7
  7. 2024How to Explore with Belief: State Entropy Maximization in POMDPscited by 6
  8. 2024The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enoughcited by 6
  9. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
  10. 2024Effective Exploration Based on the Structural Information Principlescited by 4
  11. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learningcited by 4
  12. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Storycited by 4
10 more
  1. 2023Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rolloutcited by 2
  2. 2024Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidancecited by 2
  3. 2025Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Modelscited by 2
  4. 2023Predictable Reinforcement Learning Dynamics through Entropy Rate Minimizationcited by 1
  5. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learningcited by 1
  6. 2025SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimationcited by 1
  7. 2023Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Explorationcited by 0
  8. 2025Trajectory Entropy Reinforcement Learning for Robust Robot Motor Skill Learningcited by 0
  9. 2026K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agentscited by 0
  10. 2026Global Optimality for Constrained Exploration via Penalty Regularizationcited by 0