# Optimal Control of Probabilistic Dynamics Models via Mean Hamiltonian Minimization

arXiv: https://arxiv.org/abs/2504.02543 · HTML: https://mlsanity.com/paper/2504.02543/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Sample-efficient Cross-Entropy Method for Real-time Planning](https://mlsanity.com/paper/2008.06389/) (2020)
- [Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control](https://mlsanity.com/paper/1706.06491/) (2017)
- [Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework](https://mlsanity.com/paper/1912.12970/) (2019)
- [Continuous-Time Model-Based Reinforcement Learning](https://mlsanity.com/paper/2102.04764/) (2021)
- [Efficient Exploration in Continuous-time Model-based Reinforcement Learning](https://mlsanity.com/paper/2310.19848/) (2023)
- [A Pontryagin Perspective on Reinforcement Learning](https://mlsanity.com/paper/2405.18100/) (2024)
- [Optimal Control of ensembles of dynamical systems](https://mlsanity.com/paper/2203.16485/) (2022)
- [Necessary Optimality Conditions For Average Cost Minimization Problems](https://mlsanity.com/paper/1901.04213/) (2019)
- [Convergence results for an averaged LQR problem with applications to reinforcement learning](https://mlsanity.com/paper/2011.03447/) (2020)

## Built on by (most cited first, 3 of 3)

- [Automated Discovery of Laser Dicing Processes with Bayesian Optimization for Semiconductor Manufacturing](https://mlsanity.com/paper/2511.23141/) (2025)
- [Hidden States as Value Gradients: The Pontryagin Structure of Recurrent Policies](https://mlsanity.com/paper/2605.05373/) (2026)
- [Policy Gradient for Continuous-Time Robust Markov Decision Processes](https://mlsanity.com/paper/2606.04335/) (2026)

## Abstract (arXiv)

Without exact knowledge of the true system dynamics, optimal control of non-linear continuous-time systems requires careful treatment under epistemic uncertainty. In this work, we translate a probabilistic interpretation of the Pontryagin maximum principle to the challenge of optimal control with learned probabilistic dynamics models. Our framework provides a principled treatment of epistemic uncertainty by minimizing the mean Hamiltonian with respect to a posterior distribution over the system dynamics. We propose a multiple shooting numerical method that leverages mean Hamiltonian minimization and is scalable to large-scale probabilistic dynamics models, including ensemble neural ordinary differential equations. Comparisons against other baselines in online and offline model-based reinforcement learning tasks show that our probabilistic Hamiltonian approach leads to reduced trial costs in offline settings and achieves competitive performance in online scenarios. By bridging optimal control and reinforcement learning, our approach offers a principled and practical framework for controlling uncertain systems with learned dynamics.