mlsanityData to 7 Oct 2026

From Optimization to Control: Quasi Policy Iteration

arXiv 2311.11166 ↗First posted 18 Nov 2023math.OC cs.LG cs.SYcited by 2

Not on the map. Why?

Abstract (arXiv)

Recent control algorithms for Markov decision processes (MDPs) have been designed using an implicit analogy with well-established optimization algorithms. In this paper, we adopt the quasi-Newton method (QNM) from convex optimization to introduce a novel control algorithm coined as quasi-policy iteration (QPI). In particular, QPI is based on a novel approximation of the ``Hessian'' matrix in the policy iteration algorithm, which exploits two linear structural constraints specific to MDPs and allows for the incorporation of prior information on the transition probability kernel. While the proposed algorithm has the same computational complexity as value iteration, it exhibits an empirical convergence behavior similar to that of QNM with a low sensitivity to the discount factor.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 20 of its 72 references that are in the map. First posted on arXiv 18 Nov 2023.

Builds on

20 of 72 references.

References · 20

  1. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  2. 2014Convex Optimization: Algorithms and Complexitycited by 569
  3. 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 134
  4. 2018On the insufficiency of existing momentum schemes for Stochastic Optimizationcited by 46
  5. 2016Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimizationcited by 34
  6. 2022On Almost Sure Convergence Rates of Stochastic Gradient Methodscited by 32
  7. 2018Globally Convergent Type-I Anderson Acceleration for Non-Smooth Fixed-Point Iterationscited by 24
  8. 2021Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Controlcited by 22
  9. 2019A First-Order Approach To Accelerated Value Iterationcited by 16
  10. 2018Anderson Acceleration for Reinforcement Learningcited by 14
  11. 2023Accelerating Value Iteration with Anchoringcited by 10
  12. 2021From Convex Optimization to MDPs: A Review of First-Order, Second-Order and Quasi-Newton Methods for MDPscited by 8
8 more
  1. 2022Operator Splitting Value Iterationcited by 8
  2. 2021Damped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilizationcited by 7
  3. 2022Dynamic Programming Through the Lens of Semismooth Newton-Type Methods (Extended Version)cited by 5
  4. 2017From Infinite to Finite Programs: Explicit Error Bounds with Applications to Approximate Dynamic Programmingcited by 4
  5. 2019Generalized Second Order Value Iteration in Markov Decision Processescited by 4
  6. 2020Fast Approximate Dynamic Programming for Input-Affine Dynamicscited by 3
  7. 2021Fast Approximate Dynamic Programming for Infinite-Horizon Markov Decision Processescited by 2
  8. 2019On Connections between Constrained Optimization and Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Rank-One Modified Value Iterationcited by 2
  2. 2026Limiting-Kernel Q(λ): Bridging Short and Long Horizonscited by 0