mlsanityData to 14 Oct 2026

Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures

arXiv 2412.06655 ↗First posted 9 Dec 2024cs.LG stat.MLcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Maximum entropy reinforcement learning integrates exploration into policy learning by providing additional intrinsic rewards proportional to the entropy of some distribution. In this paper, we propose a novel approach in which the intrinsic reward function is the relative entropy of the discounted distribution of states and actions (or features derived from these states and actions) visited during future time steps. This approach is motivated by two results. First, a policy maximizing the expected discounted sum of intrinsic rewards also maximizes a lower bound on the state-action value function of the decision process. Second, the distribution used in the intrinsic reward definition is the fixed point of a contraction operator. Existing algorithms can therefore be adapted to learn this fixed point off-policy and to compute the intrinsic rewards. We finally introduce an algorithm maximizing our new objective, and we show that resulting policies have good state-action space coverage and achieve high-performance control.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 37 of its 53 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

37 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Behind the Myth of Exploration in Policy Gradients31 Jan

Other references · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  8. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  9. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  10. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  11. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  12. 2018Provably Efficient Maximum Entropy Explorationcited by 242
24 more
  1. 2019Efficient Exploration via State Marginal Matchingcited by 188
  2. 2021Reinforcement Learning with Prototypical Representationscited by 179
  3. 2018Understanding the impact of entropy on policy optimizationcited by 176
  4. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  5. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  6. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  7. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  8. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 72
  9. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 45
  10. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  11. 2023Fast Rates for Maximum Entropy Explorationcited by 33
  12. 2021Geometric Entropic Explorationcited by 32
  13. 2021Regularized Policies are Reward Robustcited by 32
  14. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  15. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  16. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  17. 2019Marginalized State Distribution Entropy Regularization in Policy Optimizationcited by 16
  18. 2020A Deep Reinforcement Learning Framework for Continuous Intraday Market Biddingcited by 15
  19. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
  20. 2023TOM: Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition Occupancy Matchingcited by 10
  21. 2023Value function estimation using conditional diffusion models for controlcited by 10
  22. 2023Accelerating exploration and representation learning with offline pre-trainingcited by 7
  23. 2023Policy Gradient Algorithms Implicitly Optimize by Continuationcited by 4
  24. 2024Optimal Control of Renewable Energy Communities subject to Network Peak Fees with Model Predictive Control and Reinforcement Learning Algorithmscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
  2. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learningcited by 4
  3. 2026Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilitiescited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measures
  2. 2020Exploration by Maximizing R'enyi Entropy for Reward-Free RL Framework
  3. 2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL
  4. 2025SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation
  5. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  6. 2022Weighted Maximum Entropy Inverse Reinforcement Learning
  7. 2019Generalized Maximum Causal Entropy for Inverse Reinforcement Learning
  8. 2024ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy