mlsanityData to 7 Oct 2026

Hyperbolic Discounting and Learning over Multiple Horizons

arXiv 1902.06865 ↗First posted 19 Feb 2019stat.ML cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences. In this work we revisit the fundamentals of discounting in RL and bridge this disconnect by implementing an RL agent that acts via hyperbolic discounting. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over a strong value-based RL agent, Rainbow.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 13; 2022: 16; 2023: 11; 2024: 8; 2025: 3; 2026: 6.

58 papers.

Sort:

By area:

  1. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  2. 2021On the Expressivity of Markov Rewardcited by 71
  3. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  4. 2022Models of human preference for learning reward functionscited by 54
  5. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  6. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  7. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  8. 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
  9. 2022Settling the Reward Hypothesiscited by 33
  10. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 28
  11. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  12. 2022On the Role of Discount Factor in Offline Reinforcement Learningcited by 16
38 more
  1. 2021C-Planning: An Automatic Curriculum for Learning Goal-Reaching Taskscited by 15
  2. 2024Three Dogmas of Reinforcement Learningcited by 14
  3. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  4. 2022A Survey on Reinforcement Learning Methods in Character Animationcited by 12
  5. 2023Bootstrapped Representations in Reinforcement Learningcited by 11
  6. 2023Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewardscited by 8
  7. 2021Theoretical Guarantees of Fictitious Discount Algorithms for Episodic Reinforcement Learning and Global Convergence of Policy Gradient Methodscited by 7
  8. 2022Reinforcement Learning with Non-Exponential Discountingcited by 7
  9. 2022A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspacescited by 7
  10. 2021Learning State Representations from Random Deep Action-conditional Predictionscited by 6
  11. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  12. 2024The Curse of Diversity in Ensemble-Based Explorationcited by 6
  13. 2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
  14. 2022Best-Response Bayesian Reinforcement Learning with Bayes-adaptive POMDPs for Centaurscited by 5
  15. 2022Learning to Transfer Role Assignment Across Team Sizescited by 5
  16. 2021Taylor Expansion of Discount Factorscited by 4
  17. 2023Online Reinforcement Learning with Uncertain Episode Lengthscited by 4
  18. 2023A Tale of Sampling and Estimation in Discounted Reinforcement Learningcited by 4
  19. 2023UGAE: A Novel Approach to Non-exponential Discountingcited by 3
  20. 2023Towards a Better Understanding of Representation Dynamics under TD-learningcited by 3
  21. 2025The Formalism-Implementation Gap in Reinforcement Learning Researchcited by 3
  22. 2026SVL: Goal-Conditioned Reinforcement Learning as Survival Learningcited by 3
  23. 2021Adapting to Reward Progressivity via Spectral Reinforcement Learningcited by 2
  24. 2022Limited or Biased: Modeling Sub-Rational Human Investors in Financial Marketscited by 2
  25. 2024Reinforcement Learning Interventions on Boundedly Rational Human Agents in Frictionful Taskscited by 2
  26. 2024Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learningcited by 2
  27. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  28. 2021Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learningcited by 1
  29. 2022Intelligent Autonomous Intersection Managementcited by 1
  30. 2022Streaming Inference for Infinite Non-Stationary Clusteringcited by 1
  31. 2026Inverting the Bellman Equation: From Q-Values to World Modelscited by 1
  32. 2020Learn to Exceed: Stereo Inverse Reinforcement Learning with Concurrent Policy Optimizationcited by 0
  33. 2022Overcoming Temptation: Incentive Design For Intertemporal Choicecited by 0
  34. 2022Empirical investigations on WVA structural issuescited by 0
  35. 2022Factors of Influence of the Overestimation Bias of Q-Learningcited by 0
  36. 2022Correlation of the importances of neural network weights calculated by modern methods of overcoming catastrophic forgettingcited by 0
  37. 2022POMRL: No-Regret Learning-to-Plan with Increasing Horizonscited by 0
  38. 2023Markov Decision Processes with Time-Varying Geometric Discountingcited by 0

50 most cited of 58 All 58 (JSON)