# Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning

arXiv: https://arxiv.org/abs/1906.00572 · HTML: https://mlsanity.com/paper/1906.00572/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2019.

## Built on by (most cited first, 16 of 16)

- [Heuristic-Guided Reinforcement Learning](https://mlsanity.com/paper/2106.02757/) (2021)
- [On the consistency of hyper-parameter selection in value-based deep reinforcement learning](https://mlsanity.com/paper/2406.17523/) (2024)
- [Exploring through Random Curiosity with General Value Functions](https://mlsanity.com/paper/2211.10282/) (2022)
- [Improving Offline RL by Blending Heuristics](https://mlsanity.com/paper/2306.00321/) (2023)
- [Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards](https://mlsanity.com/paper/2310.00435/) (2023)
- [Faster Deep Reinforcement Learning with Slower Online Network](https://mlsanity.com/paper/2112.05848/) (2021)
- [Examining average and discounted reward optimality criteria in reinforcement learning](https://mlsanity.com/paper/2107.01348/) (2021)
- [Taylor Expansion of Discount Factors](https://mlsanity.com/paper/2106.06170/) (2021)
- [Orchestrated Value Mapping for Reinforcement Learning](https://mlsanity.com/paper/2203.07171/) (2022)
- [A Tale of Sampling and Estimation in Discounted Reinforcement Learning](https://mlsanity.com/paper/2304.05073/) (2023)
- [Adapting to Reward Progressivity via Spectral Reinforcement Learning](https://mlsanity.com/paper/2104.14138/) (2021)
- [Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimality](https://mlsanity.com/paper/2303.12785/) (2023)
- [Robust Action Gap Increasing with Clipped Advantage Learning](https://mlsanity.com/paper/2203.11677/) (2022)
- [An MRP Formulation for Supervised Learning: Generalized Temporal Difference Learning Models](https://mlsanity.com/paper/2404.15518/) (2024)
- [Factors of Influence of the Overestimation Bias of Q-Learning](https://mlsanity.com/paper/2210.05262/) (2022)
- [Learning in complex action spaces without policy gradients](https://mlsanity.com/paper/2410.06317/) (2024)

## Abstract (arXiv)

In an effort to better understand the different ways in which the discount factor affects the optimization process in reinforcement learning, we designed a set of experiments to study each effect in isolation. Our analysis reveals that the common perception that poor performance of low discount factors is caused by (too) small action-gaps requires revision. We propose an alternative hypothesis that identifies the size-difference of the action-gap across the state-space as the primary cause. We then introduce a new method that enables more homogeneous action-gaps by mapping value estimates to a logarithmic space. We prove convergence for this method under standard assumptions and demonstrate empirically that it indeed enables lower discount factors for approximate reinforcement-learning methods. This in turn allows tackling a class of reinforcement-learning problems that are challenging to solve with traditional methods.