Bridging Continuous-time LQR and Reinforcement Learning via Gradient Flow of the Bellman Error
On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization
Abstract (arXiv)
In this paper, we present a novel method for computing the optimal feedback gain of the infinite-horizon Linear Quadratic Regulator (LQR) problem via an ordinary differential equation. We introduce a novel continuous-time Bellman error, derived from the Hamilton-Jacobi-Bellman (HJB) equation, which quantifies the suboptimality of stabilizing policies and is parametrized in terms of the feedback gain. We analyze its properties, including its effective domain, smoothness, coerciveness and show the existence of a unique stationary point within the stability region. Furthermore, we derive a closed-form gradient expression of the Bellman error that induces a gradient flow. This converges to the optimal feedback and generates a unique trajectory which exclusively comprises stabilizing feedback policies. Additionally, this work advances interesting connections between LQR theory and Reinforcement Learning (RL) by redefining suboptimality of the Algebraic Riccati Equation (ARE) as a Bellman error, adapting a state-independent formulation, and leveraging Lyapunov equations to overcome the infinite-horizon challenge. We validate our method in a simulation and compare it to the state of the art.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 5 of its 31 references that are in the map. First posted on arXiv 11 Jun 2025.
Builds on
5 of 31 references.
References · 5
- 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
- 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 151
- 2019LQR through the Lens of First Order Methods: Discrete-time Casecited by 40
- 2020Policy Gradient-based Algorithms for Continuous-time Linear Quadratic Controlcited by 13
- 2019On Topological and Metrical Properties of Stabilizing Feedback Gains: the MIMO Casecited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Second-Order Policy Gradient Methods for the Linear Quadratic Regulator
- 2026Data-Driven Continuous-Time Linear Quadratic Regulator via Closed-Loop and Reinforcement Learning Parameterizations
- 2025On the Robustness of Derivative-free Methods for Linear Quadratic Regulator
- 2024Sample Complexity of the Linear Quadratic Regulator: A Reinforcement Learning Lens
- 2026Gradient Dominance in the Linear Quadratic Regulator: A Unified Analysis for Continuous-Time and Discrete-Time Systems
- 2025On the Gradient Domination of the LQG Problem
- 2025Policy Gradient Adaptive Control for the LQR: Indirect and Direct Approaches
- 2024Direct Data-Driven Discounted Infinite Horizon Linear Quadratic Regulator with Robustness Guarantees