mlsanityData to 7 Oct 2026

Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator

arXiv 1712.08642 ↗First posted 22 Dec 2017cs.LG stat.MLcited by 27

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) has been successfully used to solve many continuous control tasks. Despite its impressive results however, fundamental questions regarding the sample complexity of RL on continuous problems remain open. We study the performance of RL in this setting by considering the behavior of the Least-Squares Temporal Difference (LSTD) estimator on the classic Linear Quadratic Regulator (LQR) problem from optimal control. We give the first finite-time analysis of the number of samples needed to estimate the value function for a fixed static state-feedback policy to within varepsilon-relative error. In the process of deriving our result, we give a general characterization for when the minimum eigenvalue of the empirical covariance matrix formed along the sample path of a fast-mixing stochastic process concentrates above zero, extending a result by Koltchinskii and Mendelson in the independent covariates setting. Finally, we provide experimental evidence indicating that our analysis correctly captures the qualitative behavior of LSTD on several LQR instances.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 2; 2022: 7; 2023: 8; 2024: 5; 2025: 3; 2026: 1.

27 papers.

Sort:

By area:

  1. 2021Local policy search with Bayesian optimizationcited by 37
  2. 2019On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulatorcited by 20
  3. 2023Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Datacited by 12
  4. 2022Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guaranteescited by 9
  5. 2022Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulatorcited by 8
  6. 2022A Complete Characterization of Linear Estimators for Offline Policy Evaluationcited by 7
  7. 2021Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settingscited by 5
  8. 2022Augmented RBMLE-UCB Approach for Adaptive Control of Linear Quadratic Systemscited by 5
  9. 2023The Optimal Approximation Factors in Misspecified Off-Policy Value Function Estimationcited by 3
  10. 2022Transportation-Inequalities, Lyapunov Stability and Sampling for Dynamical Systems on Continuous State Spacecited by 2
  11. 2022Concentration Phenomenon for Random Dynamical Systems: An Operator Theoretic Approachcited by 1
  12. 2022Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-offcited by 1
15 more
  1. 2023Learning and Concentration for High Dimensional Linear Gaussians: an Invariant Subspace Approachcited by 1
  2. 2023From Spectral Theorem to Statistical Independence with Application to System Identificationcited by 1
  3. 2024Single Trajectory Conformal Predictioncited by 1
  4. 2025Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Samplingcited by 1
  5. 2025The Confusing Instance Principle for Online Linear Quadratic Controlcited by 1
  6. 2023Gauss-Newton Temporal Difference Learning with Nonlinear Function Approximationcited by 0
  7. 2023Policy Evaluation in Distributional LQRcited by 0
  8. 2023Reinforcement Learning with Partial Parametric Model Knowledgecited by 0
  9. 2023Data-Driven H-infinity Control with a Real-Time and Efficient Reinforcement Learning Algorithm: An Application to Autonomous Mobility-on-Demand Systemscited by 0
  10. 2024Distributed Policy Gradient for Linear Quadratic Networked Control with Limited Communication Rangecited by 0
  11. 2024On the Limited Representational Power of Value Functions and its Links to Statistical (In)Efficiencycited by 0
  12. 2024Coordinating Planning and Tracking in Layered Control Policies via Actor-Critic Learningcited by 0
  13. 2024Stability properties of gradient flow dynamics for the symmetric low-rank matrix factorization problemcited by 0
  14. 2025Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulatorcited by 0
  15. 2026Fundamental Limits of Man-in-the-Middle Attack Detection in Model-Free Reinforcement Learningcited by 0