# How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies

arXiv: https://arxiv.org/abs/1512.02011 · HTML: https://mlsanity.com/paper/1512.02011/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2015.

## Built on by (most cited first, 27 of 27)

- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control](https://mlsanity.com/paper/2405.16158/) (2024)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [On the consistency of hyper-parameter selection in value-based deep reinforcement learning](https://mlsanity.com/paper/2406.17523/) (2024)
- [Imitation Learning from Observation with Automatic Discount Scheduling](https://mlsanity.com/paper/2310.07433/) (2023)
- [Theoretical Guarantees of Fictitious Discount Algorithms for Episodic Reinforcement Learning and Global Convergence of Policy Gradient Methods](https://mlsanity.com/paper/2109.06362/) (2021)
- [Graph-Enhanced Policy Optimization in LLM Agent Training](https://mlsanity.com/paper/2510.26270/) (2025)
- [ACERAC: Efficient reinforcement learning in fine time discretization](https://mlsanity.com/paper/2104.04004/) (2021)
- [Taylor Expansion of Discount Factors](https://mlsanity.com/paper/2106.06170/) (2021)
- [Posterior Sampling for Continuing Environments](https://mlsanity.com/paper/2211.15931/) (2022)
- [A Tale of Sampling and Estimation in Discounted Reinforcement Learning](https://mlsanity.com/paper/2304.05073/) (2023)
- [Optimal Control of Renewable Energy Communities subject to Network Peak Fees with Model Predictive Control and Reinforcement Learning Algorithms](https://mlsanity.com/paper/2401.16321/) (2024)
- [Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning](https://mlsanity.com/paper/2606.21943/) (2026)
- [Truncating Trajectories in Monte Carlo Reinforcement Learning](https://mlsanity.com/paper/2305.04361/) (2023)
- [On shallow planning under partial observability](https://mlsanity.com/paper/2407.15820/) (2024)
- [Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting](https://mlsanity.com/paper/2602.04131/) (2026)
- [AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning](https://mlsanity.com/paper/2605.06149/) (2026)
- [Neural Network Based Model Predictive Control for an Autonomous Vehicle](https://mlsanity.com/paper/2107.14573/) (2021)
- [Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloning](https://mlsanity.com/paper/2112.08094/) (2021)
- [Optimizing the Long-Term Behaviour of Deep Reinforcement Learning for Pushing and Grasping](https://mlsanity.com/paper/2204.03487/) (2022)
- [Factors of Influence of the Overestimation Bias of Q-Learning](https://mlsanity.com/paper/2210.05262/) (2022)
- [Adaptive Discounting of Training Time Attacks](https://mlsanity.com/paper/2401.02652/) (2024)
- [Hard-Thresholding Meets Evolution Strategies in Reinforcement Learning](https://mlsanity.com/paper/2405.01615/) (2024)
- [Time-Scale Separation in Q-Learning: Extending TD(triangle) for Action-Value Function Decomposition](https://mlsanity.com/paper/2411.14019/) (2024)
- [Segmenting Action-Value Functions Over Time-Scales in SARSA via TD(Delta)](https://mlsanity.com/paper/2411.14783/) (2024)
- [Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed](https://mlsanity.com/paper/2412.10381/) (2024)
- [Learning the Supports for Categorical Critic in Reinforcement Learning](https://mlsanity.com/paper/2607.01880/) (2026)

## Abstract (arXiv)

Using deep neural nets as function approximator for reinforcement learning tasks have recently been shown to be very powerful for solving problems approaching real-world complexity. Using these results as a benchmark, we discuss the role that the discount factor may play in the quality of the learning process of a deep Q-network (DQN). When the discount factor progressively increases up to its final value, we empirically show that it is possible to significantly reduce the number of learning steps. When used in conjunction with a varying learning rate, we empirically show that it outperforms original DQN on several experiments. We relate this phenomenon with the instabilities of neural networks when they are used in an approximate Dynamic Programming setting. We also describe the possibility to fall within a local optimum during the learning process, thus connecting our discussion with the exploration/exploitation dilemma.