mlsanityData to 7 Oct 2026

How to Explore with Belief: State Entropy Maximization in POMDPs

arXiv 2406.02295 ↗First posted 4 Jun 2024cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent works have studied *state entropy maximization* in reinforcement learning, in which the agent's objective is to learn a policy inducing high entropy over states visitation (Hazan et al., 2019). They typically assume full observability of the state of the system, so that the entropy of the observations is maximized. In practice, the agent may only get *partial* observations, e.g., a robot perceiving the state of a physical space through proximity sensors and cameras. A significant mismatch between the entropy over observations and true states of the system can arise in those settings. In this paper, we address the problem of entropy maximization over the *true states* with a decision policy conditioned on partial observations *only*. The latter is a generalization of POMDPs, which is intractable in general. We develop a memory and computationally efficient *policy gradient* method to address a first-order relaxation of the objective defined on *belief* states, providing various formal characterizations of approximation gaps, the optimization landscape, and the *hallucination* problem. This paper aims to generalize state entropy maximization to more realistic domains that meet the challenges of applications.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

29 of 49 references.

References · 29

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  4. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  5. 2019Efficient Exploration via State Marginal Matchingcited by 186
  6. 2021Reinforcement Learning with Prototypical Representationscited by 179
  7. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  8. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  9. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  10. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  11. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  12. 2020Approximate information state for approximate planning and reinforcement learning in partially observed systemscited by 67
17 more
  1. 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
  2. 2018Policy Optimization via Importance Samplingcited by 47
  3. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 45
  4. 2022PAC Reinforcement Learning for Predictive State Representationscited by 42
  5. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  6. 2021Geometric Entropic Explorationcited by 32
  7. 2023Fast Rates for Maximum Entropy Explorationcited by 32
  8. 2019Active Exploration in Markov Decision Processescited by 27
  9. 2022Planning in Observable POMDPs in Quasipolynomial Timecited by 25
  10. 2022Partially Observable RL with B-Stability: Unified Structural Condition and Sharp Sample-Efficient Algorithmscited by 24
  11. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  12. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  13. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  14. 2020Active Model Estimation in Markov Decision Processescited by 20
  15. 2023Explore to Generalize in Zero-Shot RLcited by 20
  16. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  17. 2022k-Means Maximum Entropy Explorationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2024The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enoughcited by 6
  2. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
  3. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learningcited by 4
  4. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Storycited by 4
  5. 2026K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agentscited by 0
  6. 2026Global Optimality for Constrained Exploration via Penalty Regularizationcited by 0