mlsanityData to 14 Oct 2026

Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems

arXiv 1812.08305 ↗First posted 20 Dec 2018cs.LG math.OC stat.MLcited by 67

Not on the map. Why?

Cited by 24 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study derivative-free methods for policy optimization over the class of linear policies. We focus on characterizing the convergence rate of these methods when applied to linear-quadratic systems, and study various settings of driving noise and reward feedback. We show that these methods provably converge to within any pre-specified tolerance of the optimal policy with a number of zero-order evaluations that is an explicit polynomial of the error tolerance, dimension, and curvature properties of the problem. Our analysis reveals some interesting differences between the settings of additive driving noise and random initialization, as well as the settings of one-point and two-point reward feedback. Our theory is corroborated by extensive simulations of derivative-free methods on these systems. Along the way, we derive convergence rates for stochastic zero-order optimization algorithms when applied to a certain class of non-convex problems.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 14; 2023: 10; 2024: 12; 2025: 12; 2026: 10.

67 papers.

Sort:

By area:

  1. 2022Finite-time analysis of single-timescale actor-criticcited by 32
  2. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 27
  3. 2021Stabilizing Dynamical Systems via Policy Gradient Methodscited by 25
  4. 2022Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Pathcited by 20
  5. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 19
  6. 2024Meta-Learning Linear Quadratic Regulators: A Policy Gradient MAML Approach for Model-free LQRcited by 17
  7. 2021Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexitycited by 15
  8. 2021Derivative-Free Reinforcement Learning: A Reviewcited by 15
  9. 2022Learning Mixtures of Linear Dynamical Systemscited by 15
  10. 2023Fast Policy Learning for Linear Quadratic Control with Entropy Regularizationcited by 13
  11. 2021Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with √T Regretcited by 12
  12. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problemscited by 12
38 more
  1. 2023Revisiting LQR Control from the Perspective of Receding-Horizon Policy Gradientcited by 10
  2. 2023Learning the Kalman Filter with Fine-Grained Sample Complexitycited by 9
  3. 2022Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulatorcited by 8
  4. 2024Sample Complexity of the Linear Quadratic Regulator: A Reinforcement Learning Lenscited by 8
  5. 2021On the Sample Complexity of Decentralized Linear Quadratic Regulator with Partially Nested Information Structurecited by 7
  6. 2022Global Convergence of Direct Policy Search for State-Feedback H_∞ Robust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferentialcited by 7
  7. 2021Regret Analysis of Distributed Online LQR Control for Unknown LTI Systemscited by 6
  8. 2021Sample Efficient Reinforcement Learning In Continuous State Spaces: A Perspective Beyond Linearitycited by 6
  9. 2022Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value Functionscited by 6
  10. 2022Minibatch Stochastic Three Points Method for Unconstrained Smooth Minimizationcited by 6
  11. 2022Towards a Theoretical Foundation of Policy Optimization for Learning Control Policiescited by 6
  12. 2023Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithmscited by 6
  13. 2025Coreset-Based Task Selection for Sample-Efficient Meta-Reinforcement Learningcited by 6
  14. 2021Model-Free μ Synthesis via Adversarial Reinforcement Learningcited by 5
  15. 2022Rate-Optimal Online Convex Optimization in Adaptive Linear Controlcited by 5
  16. 2023Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulatorscited by 5
  17. 2021Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descentcited by 4
  18. 2023Learning Zero-Sum Linear Quadratic Games with Improved Sample Complexity and Last-Iterate Convergencecited by 4
  19. 2023Global Convergence of Receding-Horizon Policy Search in Learning Estimator Designscited by 4
  20. 2023Optimization Landscape of Policy Gradient Methods for Discrete-time Static Output Feedbackcited by 4
  21. 2024Rendering Wireless Environments Useful for Gradient Estimators: A Zero-Order Stochastic Federated Learning Methodcited by 4
  22. 2024Asynchronous Heterogeneous Linear Quadratic Regulator Designcited by 4
  23. 2026Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimatorcited by 4
  24. 2024Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspectivecited by 3
  25. 2025Learning Stabilizing Policies via an Unstable Subspace Representationcited by 3
  26. 2023Oracle Complexity Reduction for Model-free LQR: A Stochastic Variance-Reduced Policy Gradient Approachcited by 2
  27. 2024Model-Free Learning for the Linear Quadratic Regulator over Rate-Limited Channelscited by 2
  28. 2024System stabilization with policy optimization on unstable latent manifoldscited by 2
  29. 2025Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Datacited by 2
  30. 2025Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulatorcited by 2
  31. 2025Achieve Performatively Optimal Policy for Performative Reinforcement Learningcited by 2
  32. 2022Zeroth-Order Actor-Critic: An Evolutionary Framework for Sequential Decision Problemscited by 1
  33. 2022Black-Box Generalization: Stability of Zeroth-Order Learningcited by 1
  34. 2022On the Optimization Landscape of Dynamic Output Feedback: A Case Study for Linear Quadratic Regulatorcited by 1
  35. 2022Learning Decentralized Linear Quadratic Regulators with √T Regretcited by 1
  36. 2023Policy Evaluation in Distributional LQRcited by 1
  37. 2024Asynchronous Parallel Policy Gradient Methods for the Linear Quadratic Regulatorcited by 1
  38. 2025On the Robustness of Derivative-free Methods for Linear Quadratic Regulatorcited by 1

50 most cited of 67 All 67 (JSON)