# Discounted Reinforcement Learning Is Not an Optimization Problem

arXiv: https://arxiv.org/abs/1910.02140 · HTML: https://mlsanity.com/paper/1910.02140/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2019.

## Built on by (most cited first, 26 of 26)

- [On-Policy Deep Reinforcement Learning for the Average-Reward Criterion](https://mlsanity.com/paper/2106.07329/) (2021)
- [Influencing Long-Term Behavior in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2203.03535/) (2022)
- [Reward Centering](https://mlsanity.com/paper/2405.09999/) (2024)
- [Continual Learning In Environments With Polynomial Mixing Times](https://mlsanity.com/paper/2112.07066/) (2021)
- [Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards](https://mlsanity.com/paper/2310.00435/) (2023)
- [Age-Based Scheduling for Mobile Edge Computing: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2312.00279/) (2023)
- [RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning](https://mlsanity.com/paper/2408.01972/) (2024)
- [Average-Reward Soft Actor-Critic](https://mlsanity.com/paper/2501.09080/) (2025)
- [A Differential Perspective on Distributional Reinforcement Learning](https://mlsanity.com/paper/2506.03333/) (2025)
- [Lifetime policy reuse and the importance of task capacity](https://mlsanity.com/paper/2106.01741/) (2021)
- [Challenges to Solving Combinatorially Hard Long-Horizon Deep RL Tasks](https://mlsanity.com/paper/2206.01812/) (2022)
- [Posterior Sampling for Continuing Environments](https://mlsanity.com/paper/2211.15931/) (2022)
- [The Formalism-Implementation Gap in Reinforcement Learning Research](https://mlsanity.com/paper/2510.16175/) (2025)
- [Planning and Learning in Average Risk-aware MDPs](https://mlsanity.com/paper/2503.17629/) (2025)
- [Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions](https://mlsanity.com/paper/2608.06545/) (2026)
- [Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting](https://mlsanity.com/paper/2602.04131/) (2026)
- [Maximum Entropy Exploration Without the Rollouts](https://mlsanity.com/paper/2603.12325/) (2026)
- [AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning](https://mlsanity.com/paper/2605.06149/) (2026)
- [Physical Atari: A Robust and Accessible Platform for Real-time Reinforcement Learning on Robots](https://mlsanity.com/paper/2606.19357/) (2026)
- [Optimizing the Long-Term Average Reward for Continuing MDPs: A Technical Report](https://mlsanity.com/paper/2104.06139/) (2021)
- [Planning with Expectation Models for Control](https://mlsanity.com/paper/2104.08543/) (2021)
- [On the Convergence of Discounted Policy Gradient Methods](https://mlsanity.com/paper/2212.14066/) (2022)
- [Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces](https://mlsanity.com/paper/2306.01896/) (2023)
- [Reward Function Design for Crowd Simulation via Reinforcement Learning](https://mlsanity.com/paper/2309.12841/) (2023)
- [Feint Behaviors and Strategies: Formalization, Implementation and Evaluation](https://mlsanity.com/paper/2403.07932/) (2024)
- [Average Reward Reinforcement Learning for Omega-Regular and Mean-Payoff Objectives](https://mlsanity.com/paper/2505.15693/) (2025)

## Abstract (arXiv)

Discounted reinforcement learning is fundamentally incompatible with function approximation for control in continuing tasks. It is not an optimization problem in its usual formulation, so when using function approximation there is no optimal policy. We substantiate these claims, then go on to address some misconceptions about discounting and its connection to the average reward formulation. We encourage researchers to adopt rigorous optimization approaches, such as maximizing average reward, for reinforcement learning in continuing tasks.