# Policy Gradient Methods for the Cost-Constrained LQR: Strong Duality and Global Convergence

arXiv: https://arxiv.org/abs/2406.03734 · HTML: https://mlsanity.com/paper/2406.03734/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 13 of 13)

- [Introduction to Online Convex Optimization](https://mlsanity.com/paper/1909.05207/) (2019)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Constrained Reinforcement Learning Has Zero Duality Gap](https://mlsanity.com/paper/1910.13393/) (2019)
- [Risk Averse Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1904.00511/) (2019)
- [Convergent Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/1910.12156/) (2019)
- [Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergence](https://mlsanity.com/paper/1910.09496/) (2019)
- [Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games](https://mlsanity.com/paper/1906.00729/) (2019)
- [Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs](https://mlsanity.com/paper/2104.04901/) (2021)
- [Data-Enabled Policy Optimization for Direct Adaptive Learning of the LQR](https://mlsanity.com/paper/2401.14871/) (2024)
- [Risk-Constrained Linear-Quadratic Regulators](https://mlsanity.com/paper/2004.04685/) (2020)
- [Infinite-horizon Risk-constrained Linear Quadratic Regulator with Average Cost](https://mlsanity.com/paper/2103.15363/) (2021)

## Built on by (most cited first, 1 of 1)

- [Model-free LQG Control with Chance Constraints](https://mlsanity.com/paper/2605.31310/) (2026)

## Abstract (arXiv)

In safety-critical applications, reinforcement learning (RL) needs to consider safety constraints. However, theoretical understandings of constrained RL for continuous control are largely absent. As a case study, this paper presents a cost-constrained LQR formulation, where a number of LQR costs with user-defined penalty matrices are subject to constraints. To solve it, we propose a policy gradient primal-dual method to find an optimal state feedback gain. Despite the non-convexity of the cost-constrained LQR problem, we provide a constructive proof for strong duality and a geometric interpretation of an optimal multiplier set. By proving that the concave dual function is Lipschitz smooth, we further provide convergence guarantees for the PG primal-dual method. Finally, we perform simulations to validate our theoretical findings.