mlsanityData to 14 Oct 2026

Maximum-Entropy Exploration with Future State-Action Visitation Measures

arXiv 2603.18965 ↗First posted 19 Mar 2026cs.LG stat.MLcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Maximum entropy reinforcement learning motivates agents to explore states and actions to maximize the entropy of some distribution, typically by providing additional intrinsic rewards proportional to that entropy function. In this paper, we study intrinsic rewards proportional to the entropy of the discounted distribution of state-action features visited during future time steps. This approach is motivated by two results. First, we show that the expected sum of these intrinsic rewards is a lower bound on the entropy of the discounted distribution of state-action features visited in trajectories starting from the initial states, which we relate to an alternative maximum entropy objective. Second, we show that the distribution used in the intrinsic reward definition is the fixed point of a contraction operator and can therefore be estimated off-policy. Experiments highlight that the new objective often leads to improved visitation of features within individual trajectories, in exchange for slightly reduced visitation of features in expectation over different trajectories, as suggested by the lower bound. It also leads to improved convergence speed for learning exploration-only agents. Control performance remains similar across most methods on the considered benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 36 of its 51 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

36 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Behind the Myth of Exploration in Policy Gradients31 Jan

Other references · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  8. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  9. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  10. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  11. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  12. 2018Provably Efficient Maximum Entropy Explorationcited by 242
23 more
  1. 2019Efficient Exploration via State Marginal Matchingcited by 188
  2. 2021Reinforcement Learning with Prototypical Representationscited by 179
  3. 2018Understanding the impact of entropy on policy optimizationcited by 176
  4. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  5. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  6. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  7. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  8. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 45
  9. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  10. 2023Fast Rates for Maximum Entropy Explorationcited by 33
  11. 2021Geometric Entropic Explorationcited by 32
  12. 2021Regularized Policies are Reward Robustcited by 32
  13. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  14. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  15. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  16. 2019Marginalized State Distribution Entropy Regularization in Policy Optimizationcited by 16
  17. 2020A Deep Reinforcement Learning Framework for Continuous Intraday Market Biddingcited by 15
  18. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
  19. 2023TOM: Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition Occupancy Matchingcited by 10
  20. 2023Value function estimation using conditional diffusion models for controlcited by 10
  21. 2023Accelerating exploration and representation learning with offline pre-trainingcited by 7
  22. 2023Policy Gradient Algorithms Implicitly Optimize by Continuationcited by 4
  23. 2024Optimal Control of Renewable Energy Communities subject to Network Peak Fees with Model Predictive Control and Reinforcement Learning Algorithmscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures
  2. 2024ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy
  3. 2020Exploration by Maximizing R'enyi Entropy for Reward-Free RL Framework
  4. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control
  5. 2024Intrinsic Rewards for Exploration without Harm from Observational Noise: A Simulation Study Based on the Free Energy Principle
  6. 2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL
  7. 2022Weighted Maximum Entropy Inverse Reinforcement Learning
  8. 2026Maximum Entropy Exploration Without the Rollouts