# Stability-Certified On-Policy Data-Driven LQR via Recursive Learning and Policy Gradient

arXiv: https://arxiv.org/abs/2403.05367 · HTML: https://mlsanity.com/paper/2403.05367/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 64 references that are in the map.
- First posted on arXiv 8 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 25 of 25)

- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [On the Sample Complexity of the Linear Quadratic Regulator](https://mlsanity.com/paper/1710.01688/) (2017)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Naive Exploration is Optimal for Online LQR](https://mlsanity.com/paper/2001.09576/) (2020)
- [Learning Linear-Quadratic Regulators Efficiently with only √T Regret](https://mlsanity.com/paper/1902.06223/) (2019)
- [Formulas for Data-driven Control: Stabilization, Optimality and Robustness](https://mlsanity.com/paper/1903.06842/) (2019)
- [Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergence](https://mlsanity.com/paper/1910.09496/) (2019)
- [Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problem](https://mlsanity.com/paper/1912.11899/) (2019)
- [LQR through the Lens of First Order Methods: Discrete-time Case](https://mlsanity.com/paper/1907.08921/) (2019)
- [Logarithmic Regret for Learning Linear Quadratic Regulators Efficiently](https://mlsanity.com/paper/2002.08095/) (2020)
- [Safely Learning to Control the Constrained Linear Quadratic Regulator](https://mlsanity.com/paper/1809.10121/) (2018)
- [Finite-time Analysis of Approximate Policy Iteration for the Linear Quadratic Regulator](https://mlsanity.com/paper/1905.12842/) (2019)
- [Data informativity: a new perspective on data-driven analysis and control](https://mlsanity.com/paper/1908.00468/) (2019)
- [Robust Policy Iteration for Continuous-time Linear Quadratic Regulation](https://mlsanity.com/paper/2005.09528/) (2020)
- [From noisy data to feedback controllers: non-conservative design via a matrix S-lemma](https://mlsanity.com/paper/2006.00870/) (2020)
- [Bridging direct & indirect data-driven control formulations via regularizations and relaxations](https://mlsanity.com/paper/2101.01273/) (2021)
- [How are policy gradient methods affected by the limits of control?](https://mlsanity.com/paper/2206.06863/) (2022)
- [Efficient Off-Policy Q-Learning for Data-Based Discrete-Time LQR Problems](https://mlsanity.com/paper/2105.07761/) (2021)
- [Low-complexity Learning of Linear Quadratic Regulators from Noisy Data](https://mlsanity.com/paper/2005.01082/) (2020)
- [Robust data-driven state-feedback design](https://mlsanity.com/paper/1909.04314/) (2019)
- [On Topological Properties of the Set of Stabilizing Feedback Gains](https://mlsanity.com/paper/1904.08451/) (2019)
- [On the Certainty-Equivalence Approach to Direct Data-Driven LQR Design](https://mlsanity.com/paper/2109.06643/) (2021)
- [Online learning of data-driven controllers for unknown switched linear systems](https://mlsanity.com/paper/2105.11523/) (2021)
- [Data-driven Linear Quadratic Regulation via Semidefinite Programming](https://mlsanity.com/paper/1911.07767/) (2019)
- [Structured exploration in the finite horizon linear quadratic dual control problem](https://mlsanity.com/paper/1910.14492/) (2019)

## Built on by (most cited first, 1 of 1)

- [A Moreau Envelope Approach for LQR Meta-Policy Estimation](https://mlsanity.com/paper/2403.17364/) (2024)

## Abstract (arXiv)

In this paper, we investigate a data-driven framework to solve Linear Quadratic Regulator (LQR) problems when the dynamics is unknown, with the additional challenge of providing stability certificates for the overall learning and control scheme. Specifically, in the proposed on-policy learning framework, the control input is applied to the actual (unknown) linear system while iteratively optimized. We propose a learning and control procedure, termed Relearn LQR, that combines a recursive least squares method with a direct policy search based on the gradient method. The resulting scheme is analyzed by modeling it as a feedback-interconnected nonlinear dynamical system. A Lyapunov-based approach, exploiting averaging and timescale separation theories for nonlinear systems, allows us to provide formal stability guarantees for the whole interconnected scheme. The effectiveness of the proposed strategy is corroborated by numerical simulations, where Relearn LQR is deployed on an aircraft control problem, with both static and drifting parameters.