Data to 13 Oct 2026

Understanding the impact of entropy on policy optimization

arXiv 1811.11214 ↗First posted 27 Nov 2018cs.LG stat.MLcited by 172

Not on the map. Why?

Cited by 50 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Entropy regularization is commonly used to improve policy optimization in reinforcement learning. It is believed to help with exploration by encouraging the selection of more stochastic policies. In this work, we analyze this claim using new visualizations of the optimization landscape based on randomly perturbing the loss function. We first show that even with access to the exact gradient, policy optimization is difficult due to the geometry of the objective function. Then, we qualitatively show that in some environments, a policy with higher entropy can make the optimization landscape smoother, thereby connecting local optima and enabling the use of larger learning rates. This paper presents new tools for understanding the optimization landscape, shows that policy entropy serves as a regularizer, and highlights the challenge of designing general-purpose policy optimization algorithms.

Key facts. Cited by 172 papers in the map (data through 13 Oct 2026). First posted on arXiv 27 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2020: 1; 2021: 31; 2022: 23; 2023: 20; 2024: 29; 2025: 44; 2026: 24.

172 papers.

Sort:

By area:

2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 170
2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 115
2023Robust Scheduling with GFlowNetscited by 67
2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 61
2021Adversarially Guided Actor-Criticcited by 47
2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 40
2025On Entropy Control in LLM-RL Algorithmscited by 40
2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
2021Shapley Counterfactual Credits for Multi-Agent Reinforcement Learningcited by 37
2021Regularized Policies are Reward Robustcited by 32

38 more

2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Trainingcited by 32
2025On-Policy RL with Optimal Reward Baselinecited by 31
2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26
2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
2023RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learningcited by 24
2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learningcited by 23
2022Policy Optimization over General State and Action Spacescited by 22
2022Your Policy Regularizer is Secretly an Adversarycited by 21
2022A Differential Entropy Estimator for Training Neural Networkscited by 20
2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
2022Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithmcited by 17
2022PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimationcited by 16
2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
2021Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximationcited by 15
2021Variational Quantum Soft Actor-Criticcited by 14
2022Robust Policy Optimization in Deep Reinforcement Learningcited by 14
2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 14
2024EntAugment: Entropy-Driven Adaptive Data Augmentation Framework for Image Classificationcited by 13
2023Fast Policy Learning for Linear Quadratic Control with Entropy Regularizationcited by 12
2023Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Controlcited by 11
2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoningcited by 11
2021Optimization Issues in KL-Constrained Approximate Policy Iterationcited by 10
2021Safe Chance Constrained Reinforcement Learning for Batch Process Controlcited by 10
2024Learning Optimal Deterministic Policies with Stochastic Policy Gradientscited by 10
2024Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexitycited by 10
2024Entropy annealing for policy mirror descent in continuous time and spacecited by 10
2025On Rollouts in Model-Based Reinforcement Learningcited by 10
2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 10
2021An Alternate Policy Gradient Estimator for Softmax Policiescited by 9
2024Sourcerer: Sample-based Maximum Entropy Source Distribution Estimationcited by 9
2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 9
2025Exploring Exploration in Bayesian Optimizationcited by 9
2021Inducing Equilibria via Incentives: Simultaneous Design-and-Play Ensures Global Convergencecited by 8
2021Towards an Understanding of Default Policies in Multitask Policy Optimizationcited by 8
2024Recurrent Natural Policy Gradient for POMDPscited by 8
2024Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimizationcited by 8

50 most cited of 172 All 172 (JSON)