Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergence
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
We provide theoretical guarantees for convergence of discrete-time policy mirror descent with inexact advantage functions updated using temporal difference (TD) learning for entropy regularised MDPs in Polish state and action spaces. We rigorously derive sufficient conditions under which the single-loop actor-critic scheme is stable and convergent. To weaken these conditions, we introduce a variant that performs multiple TD steps per policy update and derive an explicit lower bound on the number of TD steps required to ensure stability. Finally, we establish sub-linear convergence when the number of TD steps grows logarithmically with the number of policy updates, and linear convergence when it grows linearly under a concentrability assumption.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 35 references that are in the map. First posted on arXiv 11 Feb 2026.
Builds on
20 of 35 references.
Closest ancestors · 1
Same direction, newest first
Other references · 19
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015Trust Region Policy Optimizationcited by 2,580
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
- 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
- 2018Understanding the impact of entropy on policy optimizationcited by 176
- 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
- 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
- 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 144
- 2020Mirror Descent Policy Optimizationcited by 106
- 2019Frequentist Regret Bounds for Randomized Least-Squares Value Iterationcited by 101
- 2022Mirror Descent with Relative Smoothness in Measure Spaces, with application to Sinkhorn and EMcited by 34
7 more
- 2021Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisitingcited by 28
- 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 23
- 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 21
- 2022Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithmcited by 20
- 2021Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximationcited by 16
- 2021Wasserstein Flow Meets Replicator Dynamics: A Mean-Field Analysis of Representation Learning in Actor-Criticcited by 7
- 2024Skill Transfer and Discovery for Sim-to-Real Learning: A Representation-Based Viewpointcited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Convergence of an actor-critic gradient flow for entropy regularised MDPs in general spaces
- 2026An analysis of Mirror-Descent Soft Actor-Critic
- 2025Mirror Descent Actor Critic via Bounded Advantage Learning
- 2026On the Policy Convergence of Policy Mirror Descent Methods
- 2025On the Convergence of Policy Mirror Descent with Temporal Difference Evaluation
- 2025StaQ: a Finite Memory Approach to Discrete Action Policy Mirror Descent
- 2026Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic
- 2025Policy Mirror Descent with Temporal Difference Learning: Sample Complexity under Online Markov Data