mlsanityData to 7 Oct 2026

Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guarantees

arXiv 2202.00048 ↗First posted 31 Jan 2022math.OC cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We propose a single time-scale actor-critic algorithm to solve the linear quadratic regulator (LQR) problem. A least squares temporal difference (LSTD) method is applied to the critic and a natural policy gradient method is used for the actor. We give a proof of convergence with sample complexity O(varepsilon-1 log(varepsilon-1)2). The method in the proof is applicable to general single time-scale bilevel optimization problem. We also numerically validate our theoretical results on the convergence.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 10 of its 33 references that are in the map. First posted on arXiv 31 Jan 2022.

Builds on

10 of 33 references.

References · 10

  1. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 211
  2. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 151
  3. 2017A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applicationscited by 126
  4. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 120
  5. 2021A Single-Timescale Method for Stochastic Bilevel Optimizationcited by 69
  6. 2019Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problemcited by 35
  7. 2020Single-Timescale Actor-Critic Provably Finds Globally Optimal Policycited by 34
  8. 2017Least-Squares Temporal Difference Learning for the Linear Quadratic Regulatorcited by 27
  9. 2021A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learningcited by 20
  10. 2019On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Costcited by 12

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 5; 2025: 2.

9 papers.

  1. 2022Finite-time analysis of single-timescale actor-criticcited by 30
  2. 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
  3. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 18
  4. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problemscited by 10
  5. 2022Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulatorcited by 8
  6. 2024Fast Two-Time-Scale Stochastic Gradient Method with Applications in Reinforcement Learningcited by 8
  7. 2024Sample Complexity of the Linear Quadratic Regulator: A Reinforcement Learning Lenscited by 3
  8. 2025Stabilizing Policy Gradient Methods via Reward Profilingcited by 1
  9. 2025Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulatorcited by 0