mlsanityData to 14 Oct 2026

Control and Reinforcement Learning through the Lens of Optimization: An Algorithmic Perspective

arXiv 2512.07377 ↗First posted 8 Dec 2025math.OCcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

The connection between control algorithms for Markov decision processes and optimization algorithms has been implicitly and explicitly exploited since the introduction of dynamic programming algorithm by Bellman in the 1950s. Recently, this connection has attracted a lot of attention for developing new control algorithms inspired by well-established optimization algorithms. In this paper, we make this analogy explicit across four problem classes with a unified solution characterization. This novel framework, in turn, allows for a systematic transformation of algorithms from one domain to the other. In particular, we identify equivalent optimization and control algorithms that have already been pointed out in the existing literature, but mostly in a scattered way. We also discuss the issues arising in providing theoretical convergence guarantees for these new control algorithms and provide simple yet effective techniques to solve them. The provided framework and techniques then lay out a concrete methodology for developing new convergent control algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 39 of its 107 references that are in the map. First posted on arXiv 8 Dec 2025.

Builds on

39 of 107 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Fitted Q-Iteration via Max-Plus-Linear Approximation12 Sep
  2. 2024Deflated Dynamics Value Iteration15 Jul
  3. 2024PID Accelerated Temporal Difference Algorithms11 Jul
  4. 2024Stochastic Halpern iteration in normed spaces and applications to reinforcement learning19 Mar

Other references · 35

  1. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  2. 2014Deep Learning in Neural Networks: An Overviewcited by 770
  3. 2014Convex Optimization: Algorithms and Complexitycited by 575
  4. 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 135
  5. 2021Accelerated Algorithms for Smooth Convex-Concave Minimax Problems with O(1/k2) Rate on Squared Gradient Normcited by 73
  6. 2018On the insufficiency of existing momentum schemes for Stochastic Optimizationcited by 49
  7. 2016Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimizationcited by 37
  8. 2022On Almost Sure Convergence Rates of Stochastic Gradient Methodscited by 35
  9. 2015Convergence rates of sub-sampled Newton methodscited by 33
  10. 2018Globally Convergent Type-I Anderson Acceleration for Non-Smooth Fixed-Point Iterationscited by 25
  11. 2021Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Controlcited by 25
  12. 2022Exact Optimal Accelerated Complexity for Fixed-Point Iterationscited by 25
23 more
  1. 2022From Halpern's Fixed-Point Iterations to Nesterov's Accelerated Interpretations for Root-Finding Problemscited by 20
  2. 2022Stochastic Halpern Iteration with Variance Reduction for Stochastic Monotone Inclusionscited by 19
  3. 2019A First-Order Approach To Accelerated Value Iterationcited by 17
  4. 2025Towards Weaker Variance Assumptions for Stochastic Optimizationcited by 17
  5. 2018Anderson Acceleration for Reinforcement Learningcited by 15
  6. 2025Near-Optimal Sample Complexity for MDPs via Anchoringcited by 14
  7. 2019Zap Q-Learning With Nonlinear Function Approximationcited by 13
  8. 2023Accelerating Value Iteration with Anchoringcited by 11
  9. 2019Regularized Anderson Acceleration for Off-Policy Deep Reinforcement Learningcited by 9
  10. 2019Anderson Acceleration of Proximal Gradient Methodscited by 9
  11. 2021From Convex Optimization to MDPs: A Review of First-Order, Second-Order and Quasi-Newton Methods for MDPscited by 9
  12. 2021Damped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilizationcited by 8
  13. 2024Truncated Variance Reduced Value Iterationcited by 8
  14. 2021Stochastic Anderson Mixing for Nonconvex Stochastic Optimizationcited by 7
  15. 2022Dynamic Programming Through the Lens of Semismooth Newton-Type Methods (Extended Version)cited by 7
  16. 2024Spectral Preconditioning for Gradient Methods on Graded Non-convex Functionscited by 6
  17. 2017From Infinite to Finite Programs: Explicit Error Bounds with Applications to Approximate Dynamic Programmingcited by 5
  18. 2019Generalized Second Order Value Iteration in Markov Decision Processescited by 5
  19. 2020Fast Approximate Dynamic Programming for Input-Affine Dynamicscited by 4
  20. 2023From Optimization to Control: Quasi Policy Iterationcited by 4
  21. 2021Fast Approximate Dynamic Programming for Infinite-Horizon Markov Decision Processescited by 3
  22. 2025Rank-One Modified Value Iterationcited by 3
  23. 2019On Connections between Constrained Optimization and Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms
  2. 2026Mathematical methods of reinforcement learning
  3. 2026Foundations of Reinforcement Learning and Control:Connections and New Perspectives
  4. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control
  5. 2023Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization
  6. 2026Relating Reinforcement Learning to Dynamic Programming-Based Planning
  7. 2026Markov Decision Processes of the Third Kind: Learning Distributions by Policy Gradient Descent
  8. 2025Maximize Your Diffusion: A Study into Reward Maximization and Alignment for Diffusion-based Control