# Some remarks on gradient dominance and LQR policy optimization

arXiv: https://arxiv.org/abs/2507.10452 · HTML: https://mlsanity.com/paper/2507.10452/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 13 of 13)

- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problem](https://mlsanity.com/paper/1912.11899/) (2019)
- [Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions](https://mlsanity.com/paper/1605.00405/) (2016)
- [Optimizing Static Linear Feedback: Gradient Method](https://mlsanity.com/paper/2004.09875/) (2020)
- [Sharp Analysis of Stochastic Optimization under Global Kurdyka-Lojasiewicz Inequality](https://mlsanity.com/paper/2210.01748/) (2022)
- [Robust Policy Iteration for Continuous-time Linear Quadratic Regulation](https://mlsanity.com/paper/2005.09528/) (2020)
- [Policy Gradient-based Algorithms for Continuous-time Linear Quadratic Control](https://mlsanity.com/paper/2006.09178/) (2020)
- [Remarks on input to state stability of perturbed gradient flows, motivated by model-free feedback control learning](https://mlsanity.com/paper/2108.02632/) (2021)
- [The role of convexity on saddle-point dynamics: Lyapunov function and robustness](https://mlsanity.com/paper/1608.08586/) (2016)
- [Small-Disturbance Input-to-State Stability of Perturbed Gradient Flows: Applications to LQR Problem](https://mlsanity.com/paper/2310.02930/) (2023)
- [Online Optimization of Switched LTI Systems Using Continuous-Time and Hybrid Accelerated Gradient Flows](https://mlsanity.com/paper/2008.03903/) (2020)
- [Robustness properties of a large-amplitude, high-frequency extremum seeking control scheme](https://mlsanity.com/paper/2009.14676/) (2020)
- [Input-to-State Stability of a Bilevel Proximal Gradient Descent Algorithm](https://mlsanity.com/paper/2211.10798/) (2022)

## Abstract (arXiv)

Solutions of optimization problems, including policy optimization in reinforcement learning, typically rely upon some variant of gradient descent. There has been much recent work in the machine learning, control, and optimization communities applying the Polyak-Lojasiewicz Inequality (PLI) to such problems in order to establish an exponential rate of convergence (a.k.a. ``linear convergence'' in the local-iteration language of numerical analysis) of loss functions to their minima under the gradient flow. Often, as is the case of policy iteration for the continuous-time LQR problem, this rate vanishes for large initial conditions, resulting in a mixed globally linear / locally exponential behavior. This is in sharp contrast with the discrete-time LQR problem, where there is global exponential convergence. That gap between CT and DT behaviors motivates the search for various generalized PLI-like conditions, and this talk will address that topic. Moreover, these generalizations are key to understanding the transient and asymptotic effects of errors in the estimation of the gradient, errors which might arise from adversarial attacks, wrong evaluation by an oracle, early stopping of a simulation, inaccurate and very approximate digital twins, stochastic computations (algorithm ``reproducibility''), or learning by sampling from limited data. We describe an ``input to state stability'' (ISS) analysis of this issue. The second part discusses convergence and PLI-like properties of ``linear feedforward neural networks'' in feedback control. Much of the work described here was done in collaboration with Arthur Castello B. de Oliveira, Leilei Cui, Zhong-Ping Jiang, and Milad Siami.