mlsanityData to 14 Oct 2026

Reinforcement Learning with Quasi-Hyperbolic Discounting

arXiv 2409.10583 ↗First posted 16 Sep 2024cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reinforcement learning has traditionally been studied with exponential discounting or the average reward setup, mainly due to their mathematical tractability. However, such frameworks fall short of accurately capturing human behavior, which has a bias towards immediate gratification. Quasi-Hyperbolic (QH) discounting is a simple alternative for modeling this bias. Unlike in traditional discounting, though, the optimal QH-policy, starting from some time t1, can be different to the one starting from t2. Hence, the future self of an agent, if it is naive or impatient, can deviate from the policy that is optimal at the start, leading to sub-optimal overall returns. To prevent this behavior, an alternative is to work with a policy anchored in a Markov Perfect Equilibrium (MPE). In this work, we propose the first model-free algorithm for finding an MPE. Using a two-timescale analysis, we show that, if our algorithm converges, then the limit must be an MPE. We also validate this claim numerically for the standard inventory system with stochastic demands. Our work significantly advances the practical application of reinforcement learning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 3 of its 19 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

3 of 19 references.

References · 3

  1. 2016Stochastic Recursive Inclusions in two timescales with non-additive iterate dependent Markov noisecited by 7
  2. 2022Actor-Critic or Critic-Actor? A Tale of Two Time Scalescited by 7
  3. 2015Stochastic recursive inclusion in two timescales with an application to the Lagrangian dual problemcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Teaching Precommitted Agents: Model-Free Policy Evaluation and Control in Quasi-Hyperbolic Discounted MDPscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPs
  2. 2026Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting
  3. 2026Finite-Time Analysis of Discounted Exponential-Utility Reinforcement Learning
  4. 2026Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting
  5. 2025Minimax Optimal Reinforcement Learning with Quasi-Optimism
  6. 2024Partial Identifiability in Inverse Reinforcement Learning For Agents With Non-Exponential Discounting
  7. 2026A Survey of Reinforcement Learning For Economics
  8. 2025Non-stationary and Varying-discounting Markov Decision Processes for Reinforcement Learning