# An Efficient Off-Policy Reinforcement Learning Algorithm for the Continuous-Time LQR Problem

arXiv: https://arxiv.org/abs/2303.17819 · HTML: https://mlsanity.com/paper/2303.17819/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 27 references that are in the map.
- First posted on arXiv 31 Mar 2023.

## Builds on (most cited first, 5 of 5)

- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [Formulas for Data-driven Control: Stabilization, Optimality and Robustness](https://mlsanity.com/paper/1903.06842/) (2019)
- [Data informativity: a new perspective on data-driven analysis and control](https://mlsanity.com/paper/1908.00468/) (2019)
- [Efficient Off-Policy Q-Learning for Data-Based Discrete-Time LQR Problems](https://mlsanity.com/paper/2105.07761/) (2021)
- [On a Continuous-Time Version of Willems' Lemma](https://mlsanity.com/paper/2203.03702/) (2022)

## Built on by (most cited first, 2 of 2)

- [Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data](https://mlsanity.com/paper/2502.19977/) (2025)
- [Data-Driven Continuous-Time Linear Quadratic Regulator via Closed-Loop and Reinforcement Learning Parameterizations](https://mlsanity.com/paper/2604.27922/) (2026)

## Abstract (arXiv)

In this paper, an off-policy reinforcement learning algorithm is designed to solve the continuous-time LQR problem using only input-state data measured from the system. Different from other algorithms in the literature, we propose the use of a specific persistently exciting input as the exploration signal during the data collection step. We then show that, using this persistently excited data, the solution of the matrix equation in our algorithm is guaranteed to exist and to be unique at every iteration. Convergence of the algorithm to the optimal control input is also proven. Moreover, we formulate the policy evaluation step as the solution of a Sylvester-transpose equation, which increases the efficiency of its solution. Finally, a method to determine a stabilizing policy to initialize the algorithm using only measured data is proposed.