Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulator
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
Actor-critic methods have achieved state-of-the-art performance in various challenging tasks. However, theoretical understandings of their performance remain elusive and challenging. Existing studies mostly focus on practically uncommon variants such as double-loop or two-timescale stepsize actor-critic algorithms for simplicity. These results certify local convergence on finite state- or action-space only. We push the boundary to investigate the classic single-sample single-timescale actor-critic on continuous (infinite) state-action space, where we employ the canonical linear quadratic regulator (LQR) problem as a case study. We show that the popular single-timescale actor-critic can attain an epsilon-optimal solution with an order of epsilon to -2 sample complexity for solving LQR on the demanding continuous state-action space. Our work provides new insights into the performance of single-timescale actor-critic, which further bridges the gap between theory and practice.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 39 references that are in the map. First posted on arXiv 2 May 2025.
Builds on
18 of 39 references.
References · 18
- 2015Trust Region Policy Optimizationcited by 2,580
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
- 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
- 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 159
- 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
- 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 122
- 2018Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulatorcited by 86
- 2019Learning Linear-Quadratic Regulators Efficiently with only √T Regretcited by 70
- 2018Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systemscited by 67
- 2019On the Sample Complexity of Actor-Critic Method for Reinforcement Learning with Function Approximationcited by 54
- 2020Non-asymptotic Convergence Analysis of Two Time-scale (Natural) Actor-Critic Algorithmscited by 43
- 2022Finite-time analysis of single-timescale actor-criticcited by 32
6 more
- 2017Least-Squares Temporal Difference Learning for the Linear Quadratic Regulatorcited by 30
- 2022A Small Gain Analysis of Single Timescale Actor Criticcited by 24
- 2019Finite-time Analysis of Approximate Policy Iteration for the Linear Quadratic Regulatorcited by 23
- 2021Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximationcited by 14
- 2022Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guaranteescited by 14
- 2023Optimization Landscape of Policy Gradient Methods for Discrete-time Static Output Feedbackcited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024On the Convergence of Single-Timescale Actor-Critic
- 2024Solving Time-Continuous Stochastic Optimal Control Problems: Algorithm Design and Convergence Analysis of Actor-Critic Flow
- 2024Closing the Gap: Achieving Global Convergence (Last Iterate) of Actor-Critic under Markovian Sampling with Neural Network Parametrization
- 2026A Concentration Bound for Two-Timescale Actor-Critic Algorithm
- 2024Compatible Gradient Approximations for Actor-Critic Algorithms
- 2024Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation
- 2025Finite Time Analysis of Constrained Natural Critic-Actor Algorithm with Improved Sample Complexity
- 2025Bridging Continuous-time LQR and Reinforcement Learning via Gradient Flow of the Bellman Error