Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
Most value-based and actor--critic reinforcement learning methods rely on Bellman-style recursions, yet these recursions collapse under non-exponential discounting common in human preferences and survival processes. We show the breakdown is structural: exponential discounting sits at a fragile intersection of multiplicativity and time homogeneity, and violating either property breaks standard dynamic programming. To overcome this, we propose Pontryagin-Guided Direct Policy Optimization (PG-DPO), a variational framework that abandons recursion and couples the Pontryagin Maximum Principle with Monte Carlo rollouts via an Adjoint-MC projection enforcing pointwise Hamiltonian maximization. Across multi-dimensional hyperbolic and survival-discount benchmarks, PG-DPO improves accuracy and stability where equation-driven solvers and critic-based baselines diverge.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 8 of its 25 references that are in the map. First posted on arXiv 20 May 2026.
Builds on
8 of 25 references.
Closest ancestors · 1
Same direction, newest first
Other references · 7
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2017Solving high-dimensional partial differential equations using deep learningcited by 556
- 2022Reinforcement Learning with Non-Exponential Discountingcited by 7
- 2023UGAE: A Novel Approach to Non-exponential Discountingcited by 3
- 2022A Stochastic Maximum Principle Approach for Reinforcement Learning with Parameterized Environmentcited by 1
- 2023On the Well-posedness of Hamilton-Jacobi-Bellman Equations of the Equilibrium Typecited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Teaching Precommitted Agents: Model-Free Policy Evaluation and Control in Quasi-Hyperbolic Discounted MDPs
- 2026Generalised Bellman recurrence and three dualities in sequential decision-making
- 2026Finite-Time Analysis of Discounted Exponential-Utility Reinforcement Learning
- 2026Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPs
- 2026Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition
- 2025Recursive Reward Aggregation
- 2025Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Model
- 2026AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning