mlsanityData to 7 Oct 2026

On the Optimization Landscape of Dynamic Output Feedback: A Case Study for Linear Quadratic Regulator

arXiv 2209.05042 ↗First posted 12 Sep 2022cs.LG cs.SY eess.SYcited by 1

Not on the map. Why?

Abstract (arXiv)

The convergence of policy gradient algorithms in reinforcement learning hinges on the optimization landscape of the underlying optimal control problem. Theoretical insights into these algorithms can often be acquired from analyzing those of linear quadratic control. However, most of the existing literature only considers the optimization landscape for static full-state or output feedback policies (controllers). We investigate the more challenging case of dynamic output-feedback policies for linear quadratic regulation (abbreviated as dLQR), which is prevalent in practice but has a rather complicated optimization landscape. We first show how the dLQR cost varies with the coordinate transformation of the dynamic controller and then derive the optimal transformation for a given observable stabilizing controller. At the core of our results is the uniqueness of the stationary point of dLQR when it is observable, which is in a concise form of an observer-based controller with the optimal similarity transformation. These results shed light on designing efficient algorithms for general decision-making problems with partially observed information.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 39 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

17 of 39 references.

References · 17

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 211
  4. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
  5. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  6. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  7. 2018Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systemscited by 56
  8. 2020Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizoncited by 43
  9. 2019Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergencecited by 39
  10. 2019LQR through the Lens of First Order Methods: Discrete-time Casecited by 30
  11. 2021Analysis of the Optimization Landscape of Linear Quadratic Gaussian (LQG) Controlcited by 23
  12. 2020Optimizing Static Linear Feedback: Gradient Methodcited by 19
5 more
  1. 2019Data informativity: a new perspective on data-driven analysis and controlcited by 14
  2. 2021Optimization Landscape of Gradient Descent for Discrete-time Static Output Feedbackcited by 9
  3. 2020Primal-dual Learning for the Model-free Risk-constrained Linear Quadratic Regulatorcited by 7
  4. 2019On Topological and Metrical Properties of Stabilizing Feedback Gains: the MIMO Casecited by 6
  5. 2020Policy Optimization for Markovian Jump Linear Quadratic Control: Gradient-Based Methods and Global Convergencecited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Accelerated Optimization Landscape of Linear-Quadratic Regulatorcited by 0