mlsanityData to 7 Oct 2026

Discount Factor as a Regularizer in Reinforcement Learning

arXiv 2007.02040 ↗First posted 4 Jul 2020cs.LG cs.AI stat.MLcited by 44

Not on the map. Why?

Abstract (arXiv)

Specifying a Reinforcement Learning (RL) task involves choosing a suitable planning horizon, which is typically modeled by a discount factor. It is known that applying RL algorithms with a lower discount factor can act as a regularizer, improving performance in the limited data regime. Yet the exact nature of this regularizer has not been investigated. In this work, we fill in this gap. For several Temporal-Difference (TD) learning methods, we show an explicit equivalence between using a reduced discount factor and adding an explicit regularization term to the algorithm's loss. Motivated by the equivalence, we empirically study this technique compared to standard L2 regularization by extensive experiments in discrete and continuous domains, using tabular and functional representations. Our experiments suggest the regularization effectiveness is strongly related to properties of the available data, such as size, distribution, and mixing rate.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 9; 2022: 8; 2023: 8; 2024: 6; 2025: 9; 2026: 3.

44 papers.

Sort:

By area:

  1. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 168
  2. 2021Heuristic-Guided Reinforcement Learningcited by 41
  3. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
  4. 2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 28
  5. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  6. 2022On the Role of Discount Factor in Offline Reinforcement Learningcited by 16
  7. 2023Behavior Alignment via Reward Function Optimizationcited by 16
  8. 2022A Survey on Reinforcement Learning Methods in Character Animationcited by 12
  9. 2023Imitation Learning from Observation with Automatic Discount Schedulingcited by 10
  10. 2025Binned Spectral Power Loss for Improved Prediction of Chaotic Systemscited by 10
  11. 2022Rethinking Value Function Learning for Generalization in Reinforcement Learningcited by 8
  12. 2023Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewardscited by 8
32 more
  1. 2024A Survey Analyzing Generalization in Deep Reinforcement Learningcited by 8
  2. 2021Theoretical Guarantees of Fictitious Discount Algorithms for Episodic Reinforcement Learning and Global Convergence of Policy Gradient Methodscited by 7
  3. 2021One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learningcited by 5
  4. 2022Learning to Transfer Role Assignment Across Team Sizescited by 5
  5. 2020Forward and inverse reinforcement learning sharing network weights and hyperparameterscited by 4
  6. 2021Taylor Expansion of Discount Factorscited by 4
  7. 2021Towards Automatic Actor-Critic Solutions to Continuous Controlcited by 4
  8. 2022Challenges to Solving Combinatorially Hard Long-Horizon Deep RL Taskscited by 4
  9. 2023A Tale of Sampling and Estimation in Discounted Reinforcement Learningcited by 4
  10. 2023UGAE: A Novel Approach to Non-exponential Discountingcited by 3
  11. 2023The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learningcited by 3
  12. 2025Bootstrapped Reward Shapingcited by 3
  13. 2025EgoCogNav: Cognition-aware Human Egocentric Navigationcited by 3
  14. 2021EnTRPO: Trust Region Policy Optimization Method with Entropy Regularizationcited by 2
  15. 2022Applications of Reinforcement Learning in Finance -- Trading with a Double Deep Q-Networkcited by 2
  16. 2025Broaden your SCOPE! Efficient Multi-turn Conversation Planning for LLMs with Semantic Spacecited by 2
  17. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  18. 2021Comparison and Unification of Three Regularization Methods in Batch Reinforcement Learningcited by 1
  19. 2023On the Value of Myopic Behavior in Policy Reusecited by 1
  20. 2024Oracle-Efficient Reinforcement Learning for Max Value Ensemblescited by 1
  21. 2024Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learningcited by 1
  22. 2024On shallow planning under partial observabilitycited by 1
  23. 2025A Dual-Agent Adversarial Framework for Robust Generalization in Deep Reinforcement Learningcited by 1
  24. 2025Deep Reinforcement Learning for Dynamic Algorithm Configuration: A Case Study on Optimizing OneMax with the (1+(λ,λ))-GAcited by 1
  25. 2026AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learningcited by 1
  26. 2021Active Reinforcement Learning over MDPscited by 0
  27. 2022Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Functioncited by 0
  28. 2022POMRL: No-Regret Learning-to-Plan with Increasing Horizonscited by 0
  29. 2023On the Effective Horizon of Inverse Reinforcement Learningcited by 0
  30. 2024Segmenting Action-Value Functions Over Time-Scales in SARSA via TD(Delta)cited by 0
  31. 2025A Translation of Probabilistic Event Calculus into Markov Decision Processescited by 0
  32. 2026Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discountingcited by 0