# Reinforcement Learning with Non-Exponential Discounting

arXiv: https://arxiv.org/abs/2209.13413 · HTML: https://mlsanity.com/paper/2209.13413/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 81 references that are in the map.
- First posted on arXiv 27 Sep 2022.

## Builds on (most cited first, 11 of 11)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Hyperbolic Discounting and Learning over Multiple Horizons](https://mlsanity.com/paper/1902.06865/) (2019)
- [Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls](https://mlsanity.com/paper/2010.14087/) (2020)
- [General non-linear Bellman equations](https://mlsanity.com/paper/1907.03687/) (2019)
- [HJB Optimal Feedback Control with Deep Differential Value Functions and Action Constraints](https://mlsanity.com/paper/1909.06153/) (2019)
- [Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System](https://mlsanity.com/paper/2110.11130/) (2021)
- [POMDPs in Continuous Time and Discrete Spaces](https://mlsanity.com/paper/2010.01014/) (2020)
- [A general theory of intertemporal decision-making and the perception of time](https://mlsanity.com/paper/1306.1200/) (2013)

## Built on by (most cited first, 7 of 7)

- [Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards](https://mlsanity.com/paper/2310.00435/) (2023)
- [UGAE: A Novel Approach to Non-exponential Discounting](https://mlsanity.com/paper/2302.05740/) (2023)
- [SVL: Goal-Conditioned Reinforcement Learning as Survival Learning](https://mlsanity.com/paper/2604.17551/) (2026)
- [Probabilistic inverse optimal control for non-linear partially observable systems disentangles perceptual uncertainty and behavioral costs](https://mlsanity.com/paper/2303.16698/) (2023)
- [Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting](https://mlsanity.com/paper/2602.04131/) (2026)
- [Probe: Learning Users' Personalized Projection Bias in Intertemporal Choices](https://mlsanity.com/paper/2303.06016/) (2023)
- [Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting](https://mlsanity.com/paper/2605.20996/) (2026)

## Abstract (arXiv)

Commonly in reinforcement learning (RL), rewards are discounted over time using an exponential function to model time preference, thereby bounding the expected long-term reward. In contrast, in economics and psychology, it has been shown that humans often adopt a hyperbolic discounting scheme, which is optimal when a specific task termination time distribution is assumed. In this work, we propose a theory for continuous-time model-based reinforcement learning generalized to arbitrary discount functions. This formulation covers the case in which there is a non-exponential random termination time. We derive a Hamilton-Jacobi-Bellman (HJB) equation characterizing the optimal policy and describe how it can be solved using a collocation method, which uses deep learning for function approximation. Further, we show how the inverse RL problem can be approached, in which one tries to recover properties of the discount function given decision data. We validate the applicability of our proposed approach on two simulated problems. Our approach opens the way for the analysis of human discounting in sequential decision-making tasks.