mlsanityData to 14 Oct 2026

Refined Analysis of Entropy-Regularized Actor-Critic

arXiv 2605.24357 ↗First posted 23 May 2026cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In this paper, we study the role of the critic in actor--critic for entropy-regularized, finite, discounted environments. We establish that, when the critic is exact, using the latter as a baseline is a variance-reduction method in a strong sense. In this case, actor--critic with stochastic gradients matches the sample complexity of deterministic policy gradient, reaching an ε-optimal regularized value with tildeO(log(1/ε)) samples. In practice, the critic is learned alongside the actor: the variance of the actor update is then influenced by the critic's variance and bias. Specifically, when the critic has a sufficiently small error, the variance reduction and rapid convergence are preserved. This suggests to learn the critic first, keeping it up to date after each actor update, underscoring the crucial role of accurate critic estimation in actor--critic methods.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 52 references that are in the map. First posted on arXiv 23 May 2026.

Builds on

31 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Beyond Softmax and Entropy: Convergence Rates of Policy Gradients with f-SoftArgmax Parameterization & Coupled Regularization18 Jan
  2. 2025Convergence of an actor-critic gradient flow for entropy regularised MDPs in general spaces16 Oct
  3. 2025Improving Value Estimation Critically Enhances Vanilla Policy Gradient25 May
  4. 2024On the Convergence of Single-Timescale Actor-Critic11 Oct

Other references · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2019A Theory of Regularized Markov Decision Processescited by 286
  7. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  8. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  9. 2017A unified view of entropy-regularized Markov decision processescited by 182
  10. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  11. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  12. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
15 more
  1. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 86
  2. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  3. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 71
  4. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 70
  5. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 58
  6. 2019On the Sample Complexity of Actor-Critic Method for Reinforcement Learning with Function Approximationcited by 54
  7. 2022Finite-time analysis of single-timescale actor-criticcited by 32
  8. 2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
  9. 2023Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processescited by 25
  10. 2022A Small Gain Analysis of Single Timescale Actor Criticcited by 24
  11. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 23
  12. 2022Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithmcited by 20
  13. 2023Federated Q-Learning: Linear Regret Speedup with Low Communication Costcited by 19
  14. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  15. 2025On Global Convergence Rates for Federated Softmax Policy Gradient under Heterogeneous Environmentscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026An analysis of Mirror-Descent Soft Actor-Criticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic
  2. 2025Actor-Critics Can Achieve Optimal Sample Efficiency
  3. 2026Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods
  4. 2022Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithms
  5. 2026Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning
  6. 2026Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergence
  7. 2026Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertainty
  8. 2024Compatible Gradient Approximations for Actor-Critic Algorithms