mlsanityData to 14 Oct 2026

Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning

arXiv 2510.08526 ↗First posted 9 Oct 2025cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In the pursuit of finding an optimal policy, reinforcement learning (RL) methods generally ignore the properties of learned policies apart from their expected return. Thus, even when successful, it is difficult to characterize which policies will be learned and what they will do. In this work, we present a theoretical framework for policy optimization that guarantees convergence to a particular optimal policy, via vanishing entropy regularization and a temperature decoupling gambit. Our approach realizes an interpretable, diversity-preserving optimal policy as the regularization temperature vanishes and ensures the convergence of policy derived objects--value functions and return distributions. In a particular instance of our method, for example, the realized policy samples all optimal actions uniformly. Leveraging our temperature decoupling gambit, we present an algorithm that estimates, to arbitrary accuracy, the return distribution associated to its interpretable, diversity-preserving optimal policy.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 49 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

31 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Tractable Representations for Convergent Approximation of Distributional HJB Equations7 Mar
  2. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning14 Oct
  3. 2024Foundations of Multivariate Distributional Reinforcement Learning31 Aug

Other references · 28

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  8. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  9. 2017A unified view of entropy-regularized Markov decision processescited by 182
  10. 2018Understanding the impact of entropy on policy optimizationcited by 176
  11. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  12. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
16 more
  1. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  2. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  3. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 63
  4. 2020Making Sense of Reinforcement Learning and Probabilistic Inferencecited by 30
  5. 2019The Value Function Polytope in Reinforcement Learningcited by 29
  6. 2019If MaxEnt RL is the Answer, What is the Question?cited by 27
  7. 2018VIREL: A Variational Inference Framework for Reinforcement Learningcited by 22
  8. 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 21
  9. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 20
  10. 2020Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regimecited by 17
  11. 2024A Distributional Analogue to the Successor Representationcited by 17
  12. 2022Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learningcited by 12
  13. 2023Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Controlcited by 11
  14. 2023Probabilistic Inference in Reinforcement Learning Done Rightcited by 11
  15. 2021Target Entropy Annealing for Discrete Soft Actor-Criticcited by 10
  16. 2021Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learningcited by 4
  2. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning in Real Option Models
  2. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
  3. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  4. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning
  5. 2025Global Convergence of Policy Gradient for Entropy Regularized Linear-Quadratic Control with Multiplicative Noise
  6. 2025State Entropy Regularization for Robust Reinforcement Learning
  7. 2025EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning
  8. 2026FERPO: Forward Entropy-Regularized Policy Optimization