# Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator

arXiv: https://arxiv.org/abs/1712.08642 · HTML: https://mlsanity.com/paper/1712.08642/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2017.

## Built on by (most cited first, 30 of 30)

- [Local policy search with Bayesian optimization](https://mlsanity.com/paper/2106.11899/) (2021)
- [On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator](https://mlsanity.com/paper/1901.03674/) (2019)
- [Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guarantees](https://mlsanity.com/paper/2202.00048/) (2022)
- [Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Data](https://mlsanity.com/paper/2308.04428/) (2023)
- [Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulator](https://mlsanity.com/paper/2208.08744/) (2022)
- [A Complete Characterization of Linear Estimators for Offline Policy Evaluation](https://mlsanity.com/paper/2203.04236/) (2022)
- [Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settings](https://mlsanity.com/paper/2111.00185/) (2021)
- [Augmented RBMLE-UCB Approach for Adaptive Control of Linear Quadratic Systems](https://mlsanity.com/paper/2201.10542/) (2022)
- [The Optimal Approximation Factors in Misspecified Off-Policy Value Function Estimation](https://mlsanity.com/paper/2307.13332/) (2023)
- [Transportation-Inequalities, Lyapunov Stability and Sampling for Dynamical Systems on Continuous State Space](https://mlsanity.com/paper/2205.12448/) (2022)
- [Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data](https://mlsanity.com/paper/2502.19977/) (2025)
- [Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Sampling](https://mlsanity.com/paper/2504.11645/) (2025)
- [Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulator](https://mlsanity.com/paper/2505.01041/) (2025)
- [Concentration Phenomenon for Random Dynamical Systems: An Operator Theoretic Approach](https://mlsanity.com/paper/2212.03670/) (2022)
- [Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off](https://mlsanity.com/paper/2212.08949/) (2022)
- [Policy Evaluation in Distributional LQR](https://mlsanity.com/paper/2303.13657/) (2023)
- [Learning and Concentration for High Dimensional Linear Gaussians: an Invariant Subspace Approach](https://mlsanity.com/paper/2304.01708/) (2023)
- [From Spectral Theorem to Statistical Independence with Application to System Identification](https://mlsanity.com/paper/2310.10523/) (2023)
- [Distributed Policy Gradient for Linear Quadratic Networked Control with Limited Communication Range](https://mlsanity.com/paper/2403.03055/) (2024)
- [Single Trajectory Conformal Prediction](https://mlsanity.com/paper/2406.01570/) (2024)
- [The Confusing Instance Principle for Online Linear Quadratic Control](https://mlsanity.com/paper/2510.19531/) (2025)
- [Gauss-Newton Temporal Difference Learning with Nonlinear Function Approximation](https://mlsanity.com/paper/2302.13087/) (2023)
- [Reinforcement Learning with Partial Parametric Model Knowledge](https://mlsanity.com/paper/2304.13223/) (2023)
- [Data-Driven H-infinity Control with a Real-Time and Efficient Reinforcement Learning Algorithm: An Application to Autonomous Mobility-on-Demand Systems](https://mlsanity.com/paper/2309.08880/) (2023)
- [On the Limited Representational Power of Value Functions and its Links to Statistical (In)Efficiency](https://mlsanity.com/paper/2403.07136/) (2024)
- [Coordinating Planning and Tracking in Layered Control Policies via Actor-Critic Learning](https://mlsanity.com/paper/2408.01639/) (2024)
- [Stability properties of gradient flow dynamics for the symmetric low-rank matrix factorization problem](https://mlsanity.com/paper/2411.15972/) (2024)
- [A Fully Data-Driven Value Iteration for Stochastic LQR: Convergence, Robustness and Stability](https://mlsanity.com/paper/2505.02970/) (2025)
- [Sample-Efficient Model-Free Policy Gradient Methods for Stochastic LQR via Robust Linear Regression](https://mlsanity.com/paper/2512.03764/) (2025)
- [Fundamental Limits of Man-in-the-Middle Attack Detection in Model-Free Reinforcement Learning](https://mlsanity.com/paper/2603.27592/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has been successfully used to solve many continuous control tasks. Despite its impressive results however, fundamental questions regarding the sample complexity of RL on continuous problems remain open. We study the performance of RL in this setting by considering the behavior of the Least-Squares Temporal Difference (LSTD) estimator on the classic Linear Quadratic Regulator (LQR) problem from optimal control. We give the first finite-time analysis of the number of samples needed to estimate the value function for a fixed static state-feedback policy to within varepsilon-relative error. In the process of deriving our result, we give a general characterization for when the minimum eigenvalue of the empirical covariance matrix formed along the sample path of a fast-mixing stochastic process concentrates above zero, extending a result by Koltchinskii and Mendelson in the independent covariates setting. Finally, we provide experimental evidence indicating that our analysis correctly captures the qualitative behavior of LSTD on several LQR instances.