mlsanityData to 7 Oct 2026

Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL

arXiv 2511.11592 ↗First posted 25 Oct 2025cs.LG cs.AI stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

Maximum entropy has become a mainstream off-policy reinforcement learning (RL) framework for balancing exploitation and exploration. However, two bottlenecks still limit further performance improvement: (1) non-stationary Q-value estimation caused by jointly injecting entropy and updating its weighting parameter, i.e., temperature; and (2) short-sighted local entropy tuning that adjusts temperature only according to the current single-step entropy, without considering the effect of cumulative entropy over time. In this paper, we extends maximum entropy framework by proposing a trajectory entropy-constrained reinforcement learning (TECRL) framework to address these two challenges. Within this framework, we first separately learn two Q-functions, one associated with reward and the other with entropy, ensuring clean and stable value targets unaffected by temperature updates. Then, the dedicated entropy Q-function, explicitly quantifying the expected cumulative entropy, enables us to enforce a trajectory entropy constraint and consequently control the policy long-term stochasticity. Building on this TECRL framework, we develop a practical off-policy algorithm, DSAC-E, by extending the state-of-the-art distributional soft actor-critic with three refinements (DSAC-T). Empirical results on the OpenAI Gym benchmark demonstrate that our DSAC-E can achieve higher returns and better stability.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 25 Oct 2025.

Builds on

29 of 36 references.

References · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  10. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  11. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  12. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
17 more
  1. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  2. 2017Parameter Space Noise for Explorationcited by 158
  3. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  4. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
  5. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
  6. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  7. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 81
  8. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  9. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  10. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
  11. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  12. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
  13. 2020Learning Off-Policy with Online Planningcited by 46
  14. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 32
  15. 2023Distributional Soft Actor-Critic with Three Refinementscited by 22
  16. 2022Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performancecited by 21
  17. 2025Enhanced DACER Algorithm with High Diffusion Efficiencycited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewardscited by 5
  2. 2026SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropycited by 0