mlsanityData to 14 Oct 2026

Fast Policy Learning for Linear Quadratic Control with Entropy Regularization

arXiv 2311.14168 ↗First posted 23 Nov 2023math.OC cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

This paper proposes and analyzes two new policy learning methods: regularized policy gradient (RPG) and iterative policy optimization (IPO), for a class of discounted linear-quadratic control (LQC) problems over an infinite time horizon with entropy regularization. Assuming access to the exact policy evaluation, both proposed approaches are proven to converge linearly in finding optimal policies of the regularized LQC. Moreover, the IPO method can achieve a super-linear convergence rate once it enters a local region around the optimal policy. Finally, when the optimal policy for an RL problem with a known environment is appropriately transferred as the initial policy to an RL problem with an unknown environment, the IPO method is shown to enable a super-linear convergence rate if the two environments are sufficiently close. Performances of these proposed algorithms are supported by numerical examples.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 23 Nov 2023.

Builds on

37 of 53 references.

References · 37

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  3. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 265
  4. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  5. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  6. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  7. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  8. 2018Understanding the impact of entropy on policy optimizationcited by 176
  9. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  10. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  11. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  12. 2021Recent Advances in Reinforcement Learning in Financecited by 110
25 more
  1. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  2. 2022q-Learning in Continuous Timecited by 91
  3. 2018Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systemscited by 67
  4. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  5. 2020Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizoncited by 50
  6. 2019Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergencecited by 46
  7. 2019LQR through the Lens of First Order Methods: Discrete-time Casecited by 40
  8. 2018Zeroth-order Nonconvex Stochastic Optimization: Handling Constraints, High-Dimensionality and Saddle-Pointscited by 37
  9. 2020Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizoncited by 35
  10. 2021Exploratory HJB equations and their convergencecited by 32
  11. 2019Dynamic Programming Principles for Mean-Field Controls with Learningcited by 23
  12. 2021On the Convergence of Prior-Guided Zeroth-Order Optimization Algorithmscited by 23
  13. 2021Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex modelscited by 21
  14. 2018Exploration versus exploitation in reinforcement learning: a stochastic control approachcited by 18
  15. 2019Policy-Gradient Algorithms Have No Guarantees of Convergence in Linear Quadratic Gamescited by 17
  16. 2021Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRscited by 16
  17. 2020Policy Gradient-based Algorithms for Continuous-time Linear Quadratic Controlcited by 13
  18. 2020Risk-Constrained Linear-Quadratic Regulatorscited by 8
  19. 2020On the Analysis of Model-free Methods for the Linear Quadratic Regulatorcited by 6
  20. 2022Optimal scheduling of entropy regulariser for continuous-time linear-quadratic reinforcement learningcited by 6
  21. 2023Markov α-Potential Gamescited by 6
  22. 2023Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulatorscited by 5
  23. 2023Risk of Transfer Learning and its Applications in Financecited by 5
  24. 2023Policy gradient learning methods for stochastic control with exit time and applications to share repurchase pricingcited by 3
  25. 2023Feasibility of Transfer Learning: A Mathematical Frameworkcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noisecited by 6
  2. 2025Bellman Optimality of Average-Reward Robust Markov Decision Processes with a Constant Gaincited by 5
  3. 2025Applications of Entropy in Data Analysis and Machine Learning: A Reviewcited by 3
  4. 2025Efficient Learning for Entropy-Regularized Markov Decision Processes via Multilevel Monte Carlocited by 3
  5. 2024Policy Optimization finds Nash Equilibrium in Regularized General-Sum LQ Gamescited by 2
  6. 2025On Policy Stochasticity in Mutual Information Optimal Control of Linear Systemscited by 1
  7. 2025Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQRcited by 1
  8. 2026Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Datacited by 1
  9. 2025Global Convergence of Policy Gradient for Entropy Regularized Linear-Quadratic Control with Multiplicative Noisecited by 0
  10. 2025Policy Transfer for Continuous-Time Reinforcement Learning: A (Rough) Differential Equation Approachcited by 0
  11. 2026Dynamic mean-variance portfolio selection with no-shorting constraints and unknown investment opportunity setscited by 0
  12. 2026Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Controlcited by 0
1 more
  1. 2026Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problemscited by 0