mlsanityData to 7 Oct 2026

Reward Centering

arXiv 2405.09999 ↗First posted 16 May 2024cs.LG cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

We show that discounted methods for solving continuing reinforcement learning problems can perform significantly better if they center their rewards by subtracting out the rewards' empirical average. The improvement is substantial at commonly used discount factors and increases further as the discount factor approaches one. In addition, we show that if a problem's rewards are shifted by a constant, then standard methods perform much worse, whereas methods with reward centering are unaffected. Estimating the average reward is straightforward in the on-policy setting; we propose a slightly more sophisticated method for the off-policy setting. Reward centering is a general idea, so we expect almost every reinforcement-learning algorithm to benefit by the addition of reward centering.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 14 of its 38 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

14 of 38 references.

References · 14

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2020Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learningcited by 93
  6. 2016Learning values across many orders of magnitudecited by 80
  7. 2019Stochastic approximation with cone-contractive operators: Sharp ℓ_∞-bounds for Q-learningcited by 74
  8. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  9. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
  10. 2019Discounted Reinforcement Learning Is Not an Optimization Problemcited by 25
  11. 2021Return-based Scaling: Yet Another Normalisation Trick for Deep RLcited by 22
  12. 2020Q-learning with Uniformly Bounded Variance: Large Discounting is Not a Barrier to Fast Learningcited by 13
2 more
  1. 2020Average Reward Adjusted Discounted Reinforcement Learning: Near-Blackwell-Optimal Policies for Real-World Applicationscited by 6
  2. 2024Step-size Optimization for Continual Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  2. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  3. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  4. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  5. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  6. 2025Learning from Expert Factors: Trajectory-level Reward Shaping for Formulaic Alpha Miningcited by 4
  7. 2026Reward Prediction with Factorized World Statescited by 2
  8. 2025An Empirical Study of Deep Reinforcement Learning in Continuing Taskscited by 1
  9. 2025A Differentiated Reward Method for Reinforcement Learning based Multi-Vehicle Cooperative Decision-Making Algorithmscited by 1
  10. 2025Learning Without Time-Based Embodiment Resets in Soft-Actor Criticcited by 1
  11. 2025A Robust Model-Based Approach for Continuous-Time Policy Evaluation with Unknown L'evy Process Dynamicscited by 0
  12. 2026Bayesian Preference Learning for Test-Time Steerable Reward Modelscited by 0
4 more
  1. 2026Deep Reinforcement Learning and The Tale of Two Temporal Difference Errorscited by 0
  2. 2026Reset-Free Reinforcement Learning for Real-World Agile Driving: An Empirical Studycited by 0
  3. 2026Extending Differential Temporal Difference Methods for Episodic Problemscited by 0
  4. 2026Accelerating Q-learning through Efficient Value-Sharing across Actionscited by 0