mlsanityData to 7 Oct 2026

Discounted Reinforcement Learning Is Not an Optimization Problem

arXiv 1910.02140 ↗First posted 4 Oct 2019cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

Discounted reinforcement learning is fundamentally incompatible with function approximation for control in continuing tasks. It is not an optimization problem in its usual formulation, so when using function approximation there is no optimal policy. We substantiate these claims, then go on to address some misconceptions about discounting and its connection to the average reward formulation. We encourage researchers to adopt rigorous optimization approaches, such as maximizing average reward, for reinforcement learning in continuing tasks.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 4; 2023: 4; 2024: 3; 2025: 5; 2026: 4.

25 papers.

Sort:

  1. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
  2. 2022Influencing Long-Term Behavior in Multiagent Reinforcement Learningcited by 18
  3. 2024Reward Centeringcited by 16
  4. 2021Continual Learning In Environments With Polynomial Mixing Timescited by 13
  5. 2023Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewardscited by 8
  6. 2023Age-Based Scheduling for Mobile Edge Computing: A Deep Reinforcement Learning Approachcited by 6
  7. 2025Average-Reward Soft Actor-Criticcited by 6
  8. 2024RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learningcited by 5
  9. 2025A Differential Perspective on Distributional Reinforcement Learningcited by 5
  10. 2021Lifetime policy reuse and the importance of task capacitycited by 4
  11. 2022Challenges to Solving Combinatorially Hard Long-Horizon Deep RL Taskscited by 4
  12. 2022Posterior Sampling for Continuing Environmentscited by 4
13 more
  1. 2025The Formalism-Implementation Gap in Reinforcement Learning Researchcited by 3
  2. 2025Planning and Learning in Average Risk-aware MDPscited by 2
  3. 2026Maximum Entropy Exploration Without the Rolloutscited by 1
  4. 2026AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learningcited by 1
  5. 2026Physical Atari: A Robust and Accessible Platform for Real-time Reinforcement Learning on Robotscited by 1
  6. 2021Optimizing the Long-Term Average Reward for Continuing MDPs: A Technical Reportcited by 0
  7. 2021Planning with Expectation Models for Controlcited by 0
  8. 2022On the Convergence of Discounted Policy Gradient Methodscited by 0
  9. 2023Learning to Stabilize Online Reinforcement Learning in Unbounded State Spacescited by 0
  10. 2023Reward Function Design for Crowd Simulation via Reinforcement Learningcited by 0
  11. 2024Feint Behaviors and Strategies: Formalization, Implementation and Evaluationcited by 0
  12. 2025Average Reward Reinforcement Learning for Omega-Regular and Mean-Payoff Objectivescited by 0
  13. 2026Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discountingcited by 0