# Global Convergence of Direct Policy Search for State-Feedback H_∞ Robust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferential

arXiv: https://arxiv.org/abs/2210.11577 · HTML: https://mlsanity.com/paper/2210.11577/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 84 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 30 of 30)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems](https://mlsanity.com/paper/1812.08305/) (2018)
- [Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon](https://mlsanity.com/paper/2011.10300/) (2020)
- [Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergence](https://mlsanity.com/paper/1910.09496/) (2019)
- [Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problem](https://mlsanity.com/paper/1912.11899/) (2019)
- [LQR through the Lens of First Order Methods: Discrete-time Case](https://mlsanity.com/paper/1907.08921/) (2019)
- [Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approach](https://mlsanity.com/paper/1912.09135/) (2019)
- [Analysis of the Optimization Landscape of Linear Quadratic Gaussian (LQG) Control](https://mlsanity.com/paper/2102.04393/) (2021)
- [Optimizing Static Linear Feedback: Gradient Method](https://mlsanity.com/paper/2004.09875/) (2020)
- [Stabilizing Dynamical Systems via Policy Gradient Methods](https://mlsanity.com/paper/2110.06418/) (2021)
- [Gradient Sampling Methods for Nonsmooth Optimization](https://mlsanity.com/paper/1804.11003/) (2018)
- [Learning the Globally Optimal Distributed LQ Regulator](https://mlsanity.com/paper/1912.08774/) (2019)
- [On the Optimization Landscape of Dynamic Output Feedback Linear Quadratic Control](https://mlsanity.com/paper/2201.09598/) (2022)
- [Optimization Landscape of Gradient Descent for Discrete-time Static Output Feedback](https://mlsanity.com/paper/2109.13132/) (2021)
- [Primal-dual Learning for the Model-free Risk-constrained Linear Quadratic Regulator](https://mlsanity.com/paper/2011.10931/) (2020)
- [Globally Convergent Policy Search over Dynamic Filters for Output Estimation](https://mlsanity.com/paper/2202.11659/) (2022)
- [On Topological and Metrical Properties of Stabilizing Feedback Gains: the MIMO Case](https://mlsanity.com/paper/1904.02737/) (2019)
- [Convergence Guarantees of Policy Optimization Methods for Markovian Jump Linear Systems](https://mlsanity.com/paper/2002.04090/) (2020)
- [Policy Learning of MDPs with Mixed Continuous/Discrete Variables: A Case Study on Model-Free Control of Markovian Jump Systems](https://mlsanity.com/paper/2006.03116/) (2020)
- [Connectivity of the Feasible and Sublevel Sets of Dynamic Output Feedback Control with Robustness Constraints](https://mlsanity.com/paper/2203.11177/) (2022)
- [Model-Free μ Synthesis via Adversarial Reinforcement Learning](https://mlsanity.com/paper/2111.15537/) (2021)
- [Escaping High-order Saddles in Policy Optimization for Linear Quadratic Gaussian (LQG) Control](https://mlsanity.com/paper/2204.00912/) (2022)
- [Minimax Lower Bounds for H_∞-Norm Estimation](https://mlsanity.com/paper/1809.10855/) (2018)
- [Can We Find Near-Approximately-Stationary Points of Nonsmooth Nonconvex Functions?](https://mlsanity.com/paper/2002.11962/) (2020)
- [Global Convergence Using Policy Gradient Methods for Model-free Markovian Jump Linear Quadratic Control](https://mlsanity.com/paper/2111.15228/) (2021)

## Built on by (most cited first, 7 of 7)

- [Fractal Landscapes in Policy Optimization](https://mlsanity.com/paper/2310.15418/) (2023)
- [Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms](https://mlsanity.com/paper/2311.18736/) (2023)
- [Learning Zero-Sum Linear Quadratic Games with Improved Sample Complexity and Last-Iterate Convergence](https://mlsanity.com/paper/2309.04272/) (2023)
- [Distributed Policy Gradient for Linear Quadratic Networked Control with Limited Communication Range](https://mlsanity.com/paper/2403.03055/) (2024)
- [Gradient Dominance in the Linear Quadratic Regulator: A Unified Analysis for Continuous-Time and Discrete-Time Systems](https://mlsanity.com/paper/2602.22577/) (2026)
- [Model-Free μ-Synthesis: A Nonsmooth Optimization Perspective](https://mlsanity.com/paper/2402.11654/) (2024)
- [Algorithmic Optimality Guarantees for Nonsmooth H_∞ Output-Feedback Policy Search](https://mlsanity.com/paper/2609.06197/) (2026)

## Abstract (arXiv)

Direct policy search has been widely applied in modern reinforcement learning and continuous control. However, the theoretical properties of direct policy search on nonsmooth robust control synthesis have not been fully understood. The optimal H_∞ control framework aims at designing a policy to minimize the closed-loop H_∞ norm, and is arguably the most fundamental robust control paradigm. In this work, we show that direct policy search is guaranteed to find the global solution of the robust H_∞ state-feedback control design problem. Notice that policy search for optimal H_∞ control leads to a constrained nonconvex nonsmooth optimization problem, where the nonconvex feasible set consists of all the policies stabilizing the closed-loop dynamics. We show that for this nonsmooth optimization problem, all Clarke stationary points are global minimum. Next, we identify the coerciveness of the closed-loop H_∞ objective function, and prove that all the sublevel sets of the resultant policy search problem are compact. Based on these properties, we show that Goldstein's subgradient method and its implementable variants can be guaranteed to stay in the nonconvex feasible set and eventually find the global optimal solution of the H_∞ state-feedback synthesis problem. Our work builds a new connection between nonconvex nonsmooth optimization theory and robust control, leading to an interesting global convergence result for direct policy search on optimal H_∞ synthesis.