mlsanityData to 14 Oct 2026

An analysis of Mirror-Descent Soft Actor-Critic

arXiv 2609.35466 ↗First posted 28 Sep 2026cs.LG math.OC stat.MLcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Soft Actor-Critic (SAC) is widely used for entropy-regularised reinforcement learning with continuous action spaces, and practical implementations perform only a few actor steps towards an evolving target. In this work, we prove convergence guarantees when the target policy arises from policy mirror descent and compare it with the classical Gibbs target. We derive sufficient conditions for the strong convexity and smoothness of the actor objective, characterised by the curvature of the Q-function estimate through the Legendre differential operator, and establish an O!left(N^-frac15right) best-iterate finite-time convergence rate up to actor and critic approximation errors. Moreover, the mirror-descent step size λ directly controls the target drift and hence actor tracking error, whereas the analogous Gibbs bound contains a non-vanishing tracking term.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 47 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

30 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Refined Analysis of Entropy-Regularized Actor-Critic23 May
  2. 2024On the Convergence of Single-Timescale Actor-Critic11 Oct

Other references · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  8. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  9. 2019A Theory of Regularized Markov Decision Processescited by 286
  10. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  11. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  12. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
16 more
  1. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  2. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 144
  3. 2020Mirror Descent Policy Optimizationcited by 106
  4. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 84
  5. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
  6. 2022Mirror Descent with Relative Smoothness in Measure Spaces, with application to Sinkhorn and EMcited by 34
  7. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 30
  8. 2022A Small Gain Analysis of Single Timescale Actor Criticcited by 24
  9. 2022Policy Optimization over General State and Action Spacescited by 23
  10. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 23
  11. 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 21
  12. 2022Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithmcited by 20
  13. 2021Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximationcited by 16
  14. 2021Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximationcited by 14
  15. 2022On the Hidden Biases of Policy Mirror Ascent in Continuous Action Spacescited by 11
  16. 2025Convergence of Policy Mirror Descent Beyond Compatible Function Approximationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergence
  2. 2025Mirror Descent Actor Critic via Bounded Advantage Learning
  3. 2026Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic
  4. 2025StaQ: a Finite Memory Approach to Discrete Action Policy Mirror Descent
  5. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning
  6. 2025Convergence of an actor-critic gradient flow for entropy regularised MDPs in general spaces
  7. 2026Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
  8. 2026On the Policy Convergence of Policy Mirror Descent Methods