mlsanityData to 7 Oct 2026

Taylor Expansion of Discount Factors

arXiv 2106.06170 ↗First posted 11 Jun 2021cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

In practical reinforcement learning (RL), the discount factor used for estimating value functions often differs from that used for defining the evaluation objective. In this work, we study the effect that this discrepancy of discount factors has during learning, and discover a family of objectives that interpolate value functions of two distinct discount factors. Our analysis suggests new ways for estimating value functions and performing policy optimization updates, which demonstrate empirical performance gains. This framework also leads to new insights on commonly-used deep RL heuristic modifications to policy optimization algorithms.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 25 of its 51 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

25 of 51 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2015Prioritized Experience Replaycited by 1,070
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  12. 2018Meta-Gradient Reinforcement Learningcited by 156
13 more
  1. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  2. 2017Time Limits in Reinforcement Learningcited by 72
  3. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 58
  4. 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 44
  5. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 43
  6. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
  7. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  8. 2019Is the Policy Gradient a Gradient?cited by 40
  9. 2015How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategiescited by 26
  10. 2019Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learningcited by 16
  11. 2019Separating value functions across time-scalescited by 15
  12. 2020Taylor Expansion Policy Optimizationcited by 9
  13. 2019General non-linear Bellman equationscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 0; 2025: 0; 2026: 1.

4 papers.

  1. 2023Reducing Blackwell and Average Optimality to Discounted MDPs via the Blackwell Discount Factorcited by 13
  2. 2023A Tale of Sampling and Estimation in Discounted Reinforcement Learningcited by 4
  3. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  4. 2023On the Value of Myopic Behavior in Policy Reusecited by 1