mlsanityData to 14 Oct 2026

A Pontryagin Perspective on Reinforcement Learning

arXiv 2405.18100 ↗First posted 28 May 2024cs.LG math.OCcited by 7

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reinforcement learning has traditionally focused on learning state-dependent policies to solve optimal control problems in a closed-loop fashion. In this work, we introduce the paradigm of open-loop reinforcement learning where a fixed action sequence is learned instead. We present three new algorithms: one robust model-based method and two sample-efficient model-free methods. Rather than basing our algorithms on Bellman's equation from dynamic programming, our work builds on Pontryagin's principle from the theory of open-loop optimal control. We provide convergence guarantees and evaluate all methods empirically on a pendulum swing-up task, as well as on two high-dimensional MuJoCo tasks, significantly outperforming existing baselines.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 46 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

14 of 46 references.

References · 14

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  5. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 700
  6. 2019Differentiation of Blackbox Combinatorial Solverscited by 226
  7. 2016Training Deep Spiking Neural Networks using Backpropagationcited by 205
  8. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 151
  9. 2021Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systemscited by 128
  10. 2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 95
  11. 2019Pontryagin Differentiable Programming: An End-to-End Learning and Control Frameworkcited by 52
  12. 2021Implicit energy regularization of neural ordinary-differential-equation controlcited by 29
2 more
  1. 2023GVFs in the Real World: Making Predictions Online for Water Treatmentcited by 13
  2. 2022Myriad: a real-world testbed to bridge trajectory optimization and deep learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025Optimal Control of Probabilistic Dynamics Models via Mean Hamiltonian Minimizationcited by 3
  2. 2025Continuous-Time Value Iteration for Multi-Agent Reinforcement Learningcited by 2
  3. 2026Memory-Induced Supra-Competitive Outcomes Between Deep Reinforcement Learning Agents in Optimal Trade Executioncited by 2
  4. 2026Commit to the Bit: Reactive Reinforcement Learning Done Rightcited by 1
  5. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0
  6. 2026Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discountingcited by 0
  7. 2026Physics-enhanced reinforcement learning for real-time optimal control of dynamical systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms
  2. 2026Foundations of Reinforcement Learning and Control:Connections and New Perspectives
  3. 2026Mathematical methods of reinforcement learning
  4. 2025Control and Reinforcement Learning through the Lens of Optimization: An Algorithmic Perspective
  5. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Space
  6. 2024Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control
  7. 2025On the Convergence and Stability of Upside-Down Reinforcement Learning, Goal-Conditioned Supervised Learning, and Online Decision Transformers
  8. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency