# Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting

arXiv: https://arxiv.org/abs/2605.20996 · HTML: https://mlsanity.com/paper/2605.20996/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 8 of 8)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Reinforcement Learning with Non-Exponential Discounting](https://mlsanity.com/paper/2209.13413/) (2022)
- [A Pontryagin Perspective on Reinforcement Learning](https://mlsanity.com/paper/2405.18100/) (2024)
- [UGAE: A Novel Approach to Non-exponential Discounting](https://mlsanity.com/paper/2302.05740/) (2023)
- [A Stochastic Maximum Principle Approach for Reinforcement Learning with Parameterized Environment](https://mlsanity.com/paper/2208.02241/) (2022)
- [On the Well-posedness of Hamilton-Jacobi-Bellman Equations of the Equilibrium Type](https://mlsanity.com/paper/2307.01986/) (2023)

## Abstract (arXiv)

Most value-based and actor--critic reinforcement learning methods rely on Bellman-style recursions, yet these recursions collapse under non-exponential discounting common in human preferences and survival processes. We show the breakdown is structural: exponential discounting sits at a fragile intersection of multiplicativity and time homogeneity, and violating either property breaks standard dynamic programming. To overcome this, we propose Pontryagin-Guided Direct Policy Optimization (PG-DPO), a variational framework that abandons recursion and couples the Pontryagin Maximum Principle with Monte Carlo rollouts via an Adjoint-MC projection enforcing pointwise Hamiltonian maximization. Across multi-dimensional hyperbolic and survival-discount benchmarks, PG-DPO improves accuracy and stability where equation-driven solvers and critic-based baselines diverge.