mlsanityData to 14 Oct 2026

Teaching Precommitted Agents: Model-Free Policy Evaluation and Control in Quasi-Hyperbolic Discounted MDPs

arXiv 2509.06094 ↗First posted 7 Sep 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Time-inconsistent preferences, where agents favor smaller-sooner over larger-later rewards, are a key feature of human and animal decision-making. Quasi-Hyperbolic (QH) discounting provides a simple yet powerful model for this behavior, but its integration into the reinforcement learning (RL) framework has been limited. This paper addresses key theoretical and algorithmic gaps for precommitted agents with QH preferences. We make two primary contributions: (i) we formally characterize the structure of the optimal policy, proving for the first time that it reduces to a simple one-step non-stationary form; and (ii) we design the first practical, model-free algorithms for both policy evaluation and Q-learning in this setting, both with provable convergence guarantees. Our results provide foundational insights for incorporating QH preferences in RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 2 of its 15 references that are in the map. First posted on arXiv 7 Sep 2025.

Builds on

2 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Reinforcement Learning with Quasi-Hyperbolic Discounting16 Sep

Other references · 1

  1. 2021Prospect-theoretic Q-learningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPs
  2. 2025Non-stationary and Varying-discounting Markov Decision Processes for Reinforcement Learning
  3. 2026Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting
  4. 2026Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting
  5. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learning
  6. 2026Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems
  7. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  8. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design