# Policy Gradient Adaptive Control for the LQR: Indirect and Direct Approaches

arXiv: https://arxiv.org/abs/2505.03706 · HTML: https://mlsanity.com/paper/2505.03706/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 6 May 2025.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 21 of 21)

- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [On the Sample Complexity of the Linear Quadratic Regulator](https://mlsanity.com/paper/1710.01688/) (2017)
- [Naive Exploration is Optimal for Online LQR](https://mlsanity.com/paper/2001.09576/) (2020)
- [Learning Linear-Quadratic Regulators Efficiently with only √T Regret](https://mlsanity.com/paper/1902.06223/) (2019)
- [Formulas for Data-driven Control: Stabilization, Optimality and Robustness](https://mlsanity.com/paper/1903.06842/) (2019)
- [Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problem](https://mlsanity.com/paper/1912.11899/) (2019)
- [LQR through the Lens of First Order Methods: Discrete-time Case](https://mlsanity.com/paper/1907.08921/) (2019)
- [Data-Enabled Predictive Control: In the Shallows of the DeePC](https://mlsanity.com/paper/1811.05890/) (2018)
- [Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs](https://mlsanity.com/paper/2104.04901/) (2021)
- [A Historical Perspective of Adaptive Control and Learning](https://mlsanity.com/paper/2108.11336/) (2021)
- [Data-Enabled Policy Optimization for Direct Adaptive Learning of the LQR](https://mlsanity.com/paper/2401.14871/) (2024)
- [On the Certainty-Equivalence Approach to Direct Data-Driven LQR Design](https://mlsanity.com/paper/2109.06643/) (2021)
- [The informativity approach to data-driven analysis and control](https://mlsanity.com/paper/2302.10488/) (2023)
- [Harnessing Uncertainty for a Separation Principle in Direct Data-Driven Predictive Control](https://mlsanity.com/paper/2312.14788/) (2023)
- [Regularization for Covariance Parameterization of Direct Data-Driven LQR Control](https://mlsanity.com/paper/2503.02985/) (2025)
- [Adaptive Control By Regulation-Triggered Batch Least-Squares Estimation of Non-Observable Parameters](https://mlsanity.com/paper/1811.10833/) (2018)
- [Online Algorithms and Policies Using Adaptive and Machine Learning Approaches](https://mlsanity.com/paper/2105.06577/) (2021)
- [Data-enabled Policy Optimization for the Linear Quadratic Regulator](https://mlsanity.com/paper/2303.17958/) (2023)
- [Robustness of Online Identification-based Policy Iteration to Noisy Data](https://mlsanity.com/paper/2504.07627/) (2025)
- [Direct Adaptive Control of Grid-Connected Power Converters via Output-Feedback Data-Enabled Policy Optimization](https://mlsanity.com/paper/2411.03909/) (2024)
- [Any-Time Regret-Guaranteed Algorithm for Control of Linear Quadratic Systems](https://mlsanity.com/paper/2406.07746/) (2024)

## Built on by (most cited first, 5 of 5)

- [Sample-efficient and Scalable Exploration in Continuous-Time RL](https://mlsanity.com/paper/2510.24482/) (2025)
- [Optimistic Online LQR via Intrinsic Rewards](https://mlsanity.com/paper/2603.28938/) (2026)
- [Sample-Efficient Model-Free Policy Gradient Methods for Stochastic LQR via Robust Linear Regression](https://mlsanity.com/paper/2512.03764/) (2025)
- [Adaptive Control of Unknown Linear Switched Systems via Policy Gradient Methods](https://mlsanity.com/paper/2601.03016/) (2026)
- [A Stochastic Gradient Descent Approach to Design Policy Gradient Methods for LQR](https://mlsanity.com/paper/2602.18933/) (2026)

## Abstract (arXiv)

Motivated by recent advances of reinforcement learning and direct data-driven control, we propose policy gradient adaptive control (PGAC) for the linear quadratic regulator (LQR), which uses online closed-loop data to improve the control policy while maintaining stability. Our method adaptively updates the policy in feedback by descending the gradient of the LQR cost and is categorized as indirect, when gradients are computed via an estimated model, versus direct, when gradients are derived from data using sample covariance parameterization. Beyond the vanilla gradient, we also showcase the merits of the natural gradient and Gauss-Newton methods for the policy update. Notably, natural gradient descent bridges the indirect and direct PGAC, and the Gauss-Newton method of the indirect PGAC leads to an adaptive version of the celebrated Hewer's algorithm. To account for the uncertainty from noise, we propose a regularization method for both indirect and direct PGAC. For all the considered PGAC approaches, we show closed-loop stability and convergence of the policy to the optimal LQR gain. Simulations validate our theoretical findings and demonstrate the robustness and computational efficiency of PGAC.