Data to 7 Oct 2026

A unified view of entropy-regularized Markov decision processes

arXiv 1705.07798 ↗First posted 22 May 2017cs.LG cs.AI stat.MLcited by 174

Not on the map. Why?

Abstract (arXiv)

We propose a general framework for entropy-regularized average-reward reinforcement learning in Markov decision processes (MDPs). Our approach is based on extending the linear-programming formulation of policy optimization in MDPs to accommodate convex regularization functions. Our key result is showing that using the conditional entropy of the joint state-action distributions as regularization yields a dual optimization problem closely resembling the Bellman optimality equations. This result enables us to formalize a number of state-of-the-art entropy-regularized reinforcement learning algorithms as approximate variants of Mirror Descent or Dual Averaging, and thus to argue about the convergence properties of these methods. In particular, we show that the exact version of the TRPO algorithm of Schulman et al. (2015) actually converges to the optimal policy, while the entropy-regularized policy gradient methods of Mnih et al. (2016) may fail to converge to a fixed point. Finally, we illustrate empirically the effects of using various regularization techniques on learning performance in a simple reinforcement learning setup.

Key facts. Cited by 174 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 27; 2022: 27; 2023: 25; 2024: 26; 2025: 35; 2026: 32.

174 papers.

Sort:

By area:

2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 65
2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 61
2025Efficient Online Reinforcement Learning for Diffusion Policycited by 61
2023Generative Flow Networks as Entropy-Regularized RLcited by 60

38 more

2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
2024Feedback Efficient Online Fine-Tuning of Diffusion Modelscited by 52
2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 44
2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
2021Batch size-invariance for policy optimizationcited by 33
2023Fast Rates for Maximum Entropy Explorationcited by 32
2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 29
2021Near-optimal Policy Optimization Algorithms for Learning Adversarial Linear Mixture MDPscited by 28
2021The Confluence of Networks, Games and Learningcited by 26
2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
2022Performative Reinforcement Learningcited by 25
2022Proximal Point Imitation Learningcited by 25
2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24
2021Policy Optimization for Constrained MDPs with Provable Fast Global Convergencecited by 22
2022Your Policy Regularizer is Secretly an Adversarycited by 21
2022Policy Optimization over General State and Action Spacescited by 21
2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 20
2020Average-reward model-free reinforcement learning: a systematic review and literature mappingcited by 19
2022You May Not Need Ratio Clipping in PPOcited by 19
2022Regularized Gradient Descent Ascent for Two-Player Zero-Sum Markov Gamescited by 18
2023Offline Primal-Dual Reinforcement Learning for Linear MDPscited by 17
2021A general class of surrogate functions for stable and efficient reinforcement learningcited by 16
2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learningcited by 16
2021Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximationcited by 15
2021Bregman Gradient Policy Optimizationcited by 15
2021Divergence-Regularized Multi-Agent Actor-Criticcited by 15
2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
2021Reinforcement learning of rare diffusive dynamicscited by 14
2022Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexitycited by 14
2022KL-Entropy-Regularized RL with a Generative Model is Minimax Optimalcited by 13
2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
2023Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RLcited by 13
2024How Can LLM Guide RL? A Value-Based Approachcited by 13

50 most cited of 174 All 174 (JSON)