mlsanityData to 7 Oct 2026

Examining average and discounted reward optimality criteria in reinforcement learning

arXiv 2107.01348 ↗First posted 3 Jul 2021cs.LG cs.AI cs.ROcited by 6

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning (RL), the goal is to obtain an optimal policy, for which the optimality criterion is fundamentally important. Two major optimality criteria are average and discounted rewards. While the latter is more popular, it is problematic to apply in environments without an inherent notion of discounting. This motivates us to revisit a) the progression of optimality criteria in dynamic programming, b) justification for and complication of an artificial discount factor, and c) benefits of directly maximizing the average reward criterion, which is discounting-free. Our contributions include a thorough examination of the relationship between average and discounted rewards, as well as a discussion of their pros and cons in RL. We emphasize that average-reward RL methods possess the ingredient and mechanism for applying a family of discounting-free optimality criteria (Veinott, 1969) to RL.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 16 of its 62 references that are in the map. First posted on arXiv 3 Jul 2021.

Builds on

16 of 62 references.

References · 16

  1. 2016OpenAI Gymcited by 1,684
  2. 2017Deep Reinforcement Learning that Matterscited by 727
  3. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  4. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  5. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  6. 2018Meta-Gradient Reinforcement Learningcited by 156
  7. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  8. 2017Time Limits in Reinforcement Learningcited by 72
  9. 2016Unifying task specification in reinforcement learningcited by 41
  10. 2019Is the Policy Gradient a Gradient?cited by 40
  11. 2021Towards Tight Bounds on the Sample Complexity of Average-reward MDPscited by 32
  12. 2020Average-reward model-free reinforcement learning: a systematic review and literature mappingcited by 19
4 more
  1. 2019Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learningcited by 16
  2. 2020Q-learning with Uniformly Bounded Variance: Large Discounting is Not a Barrier to Fast Learningcited by 13
  3. 2020Zeroth-order Deterministic Policy Gradientcited by 9
  4. 2018TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learningcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 3.

6 papers.

  1. 2023Reducing Blackwell and Average Optimality to Discounted MDPs via the Blackwell Discount Factorcited by 13
  2. 2023Off-Policy Average Reward Actor-Critic with Deterministic Policy Searchcited by 10
  3. 2023AutoRL Hyperparameter Landscapescited by 8
  4. 2024RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learningcited by 5
  5. 2024When Do Off-Policy and On-Policy Policy Gradient Methods Align?cited by 2
  6. 2024Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learningcited by 1