mlsanityData to 7 Oct 2026

The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enough

arXiv 2406.12795 ↗First posted 18 Jun 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

The problem of pure exploration in Markov decision processes has been cast as maximizing the entropy over the state distribution induced by the agent's policy, an objective that has been extensively studied. However, little attention has been dedicated to state entropy maximization under partial observability, despite the latter being ubiquitous in applications, e.g., finance and robotics, in which the agent only receives noisy observations of the true state governing the system's dynamics. How can we address state entropy maximization in those domains? In this paper, we study the simple approach of maximizing the entropy over observations in place of true latent states. First, we provide lower and upper bounds to the approximation of the true state entropy that only depends on some properties of the observation function. Then, we show how knowledge of the latter can be exploited to compute a principled regularization of the observation entropy to improve performance. With this work, we provide both a flexible approach to bring advances in state entropy maximization to the POMDP setting and a theoretical characterization of its intrinsic limits.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

30 of 47 references.

References · 30

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  3. 2019Efficient Exploration via State Marginal Matchingcited by 186
  4. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  5. 2021APS: Active Pretraining with Successor Featurescited by 109
  6. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  7. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  8. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  9. 2021Reward is enough for convex MDPscited by 82
  10. 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
  11. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 54
  12. 2018Policy Optimization via Importance Samplingcited by 47
18 more
  1. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 45
  2. 2022PAC Reinforcement Learning for Predictive State Representationscited by 42
  3. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 36
  4. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  5. 2021Geometric Entropic Explorationcited by 32
  6. 2023Fast Rates for Maximum Entropy Explorationcited by 32
  7. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
  8. 2022Planning in Observable POMDPs in Quasipolynomial Timecited by 25
  9. 2022Partially Observable RL with B-Stability: Unified Structural Condition and Sharp Sample-Efficient Algorithmscited by 24
  10. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  11. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  12. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  13. 2020Active Model Estimation in Markov Decision Processescited by 20
  14. 2023Explore to Generalize in Zero-Shot RLcited by 20
  15. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  16. 2022k-Means Maximum Entropy Explorationcited by 11
  17. 2024How to Explore with Belief: State Entropy Maximization in POMDPscited by 6
  18. 2021Entropy Maximization for Partially Observable Markov Decision Processescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
  2. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learningcited by 4
  3. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Storycited by 4
  4. 2025Convergence of regularized agent-state-based Q-learning in POMDPscited by 0
  5. 2026K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agentscited by 0
  6. 2026Global Optimality for Constrained Exploration via Penalty Regularizationcited by 0