mlsanityData to 14 Oct 2026

When Maximum Entropy Misleads Policy Optimization

arXiv 2506.05615 ↗First posted 5 Jun 2025cs.LG cs.AIcited by 7

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

The Maximum Entropy Reinforcement Learning (MaxEnt RL) framework is a leading approach for achieving efficient learning and robust performance across many RL tasks. However, MaxEnt methods have also been shown to struggle with performance-critical control problems in practice, where non-MaxEnt algorithms can successfully learn. In this work, we analyze how the trade-off between robustness and optimality affects the performance of MaxEnt algorithms in complex control tasks: while entropy maximization enhances exploration and robustness, it can also mislead policy optimization, leading to failure in tasks that require precise, low-entropy policies. Through experiments on a variety of control problems, we concretely demonstrate this misleading effect. Our analysis leads to better understanding of how to balance reward design and entropy maximization in challenging control problems.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 48 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

27 of 48 references.

References · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2016OpenAI Gymcited by 1,736
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  9. 2018Robustness May Be at Odds with Accuracycited by 880
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  11. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
  12. 2018Maximum a Posteriori Policy Optimisationcited by 328
15 more
  1. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  2. 2018Understanding the impact of entropy on policy optimizationcited by 176
  3. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  4. 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 149
  5. 2018Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Modelscited by 135
  6. 2019Adversarial Training Can Hurt Generalizationcited by 132
  7. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  8. 2024Humanoid Parkour Learningcited by 96
  9. 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 46
  10. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  11. 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flowcited by 26
  12. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learningcited by 17
  13. 2020Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradientcited by 14
  14. 2021Deep Reinforcement Learning-based UAV Navigation and Control: A Soft Actor-Critic with Hindsight Experience Replay Approachcited by 4
  15. 2022Performance Comparison of Deep RL Algorithms for Energy Systems Optimal Schedulingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 52
  2. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoningcited by 11
  3. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimizationcited by 3
  4. 2025Coordinated Strategies in Realistic Air Combat by Hierarchical Multi-Agent Reinforcement Learningcited by 1
  5. 2025Failure Modes of Maximum Entropy RLHFcited by 0
  6. 2025Complexity-Regularized Proximal Policy Optimizationcited by 0
  7. 2025Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation
  2. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  3. 2025Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularization
  4. 2025Maximum Entropy Reinforcement Learning with Diffusion Policy
  5. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learning
  6. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control
  7. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures
  8. 2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL