mlsanityData to 14 Oct 2026

Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation

arXiv 2407.18143 ↗First posted 25 Jul 2024cs.LG cs.AIcited by 4

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Entropy Regularisation is a widely adopted technique that enhances policy optimisation performance and stability. A notable form of entropy regularisation is augmenting the objective with an entropy term, thereby simultaneously optimising the expected return and the entropy. This framework, known as maximum entropy reinforcement learning (MaxEnt RL), has shown theoretical and empirical successes. However, its practical application in straightforward on-policy actor-critic settings remains surprisingly underexplored. We hypothesise that this is due to the difficulty of managing the entropy reward in practice. This paper proposes a simple method of separating the entropy objective from the MaxEnt RL objective, which facilitates the implementation of MaxEnt RL in on-policy settings. Our empirical evaluations demonstrate that extending Proximal Policy Optimisation (PPO) and Trust Region Policy Optimisation (TRPO) within the MaxEnt framework improves policy optimisation performance in both MuJoCo and Procgen tasks. Additionally, our results highlight MaxEnt RL's capacity to enhance generalisation.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 35 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

27 of 35 references.

References · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  7. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  8. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  9. 2019A Theory of Regularized Markov Decision Processescited by 286
  10. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  11. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  12. 2017A unified view of entropy-regularized Markov decision processescited by 182
15 more
  1. 2018Understanding the impact of entropy on policy optimizationcited by 176
  2. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  3. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  4. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  5. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
  6. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  7. 2020Mirror Descent Policy Optimizationcited by 106
  8. 2016Learning values across many orders of magnitudecited by 85
  9. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 68
  10. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62
  11. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  12. 2023Fast Rates for Maximum Entropy Explorationcited by 33
  13. 2019If MaxEnt RL is the Answer, What is the Question?cited by 27
  14. 2019Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learningcited by 20
  15. 2022Do You Need the Entropy Reward (in Practice)?cited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraintscited by 4
  2. 2025Reinforcement Learning for Robust Athletic Intelligence: Lessons from the 2nd 'AI Olympics with RealAIGym' Competitioncited by 2
  3. 2024Average-Reward Maximum Entropy Reinforcement Learning for Underactuated Double Pendulum Taskscited by 1
  4. 2026Exploration-Preserving Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When Maximum Entropy Misleads Policy Optimization
  2. 2025Maximum Entropy Reinforcement Learning with Diffusion Policy
  3. 2025Failure Modes of Maximum Entropy RLHF
  4. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  5. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  6. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures
  7. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learning
  8. 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow