# A Pontryagin Perspective on Reinforcement Learning

arXiv: https://arxiv.org/abs/2405.18100 · HTML: https://mlsanity.com/paper/2405.18100/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 46 references that are in the map.
- First posted on arXiv 28 May 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 14 of 14)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Differentiation of Blackbox Combinatorial Solvers](https://mlsanity.com/paper/1912.02175/) (2019)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [Sample-efficient Cross-Entropy Method for Real-time Planning](https://mlsanity.com/paper/2008.06389/) (2020)
- [Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework](https://mlsanity.com/paper/1912.12970/) (2019)
- [Implicit energy regularization of neural ordinary-differential-equation control](https://mlsanity.com/paper/2103.06525/) (2021)
- [GVFs in the Real World: Making Predictions Online for Water Treatment](https://mlsanity.com/paper/2312.01624/) (2023)
- [Myriad: a real-world testbed to bridge trajectory optimization and deep learning](https://mlsanity.com/paper/2202.10600/) (2022)

## Built on by (most cited first, 7 of 7)

- [Optimal Control of Probabilistic Dynamics Models via Mean Hamiltonian Minimization](https://mlsanity.com/paper/2504.02543/) (2025)
- [Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.09135/) (2025)
- [Memory-Induced Supra-Competitive Outcomes Between Deep Reinforcement Learning Agents in Optimal Trade Execution](https://mlsanity.com/paper/2605.20348/) (2026)
- [Commit to the Bit: Reactive Reinforcement Learning Done Right](https://mlsanity.com/paper/2605.28276/) (2026)
- [A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic](https://mlsanity.com/paper/2603.28971/) (2026)
- [Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting](https://mlsanity.com/paper/2605.20996/) (2026)
- [Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems](https://mlsanity.com/paper/2607.16177/) (2026)

## Abstract (arXiv)

Reinforcement learning has traditionally focused on learning state-dependent policies to solve optimal control problems in a closed-loop fashion. In this work, we introduce the paradigm of open-loop reinforcement learning where a fixed action sequence is learned instead. We present three new algorithms: one robust model-based method and two sample-efficient model-free methods. Rather than basing our algorithms on Bellman's equation from dynamic programming, our work builds on Pontryagin's principle from the theory of open-loop optimal control. We provide convergence guarantees and evaluate all methods empirically on a pendulum swing-up task, as well as on two high-dimensional MuJoCo tasks, significantly outperforming existing baselines.