mlsanityData to 7 Oct 2026

Return-based Scaling: Yet Another Normalisation Trick for Deep RL

arXiv 2105.05347 ↗First posted 11 May 2021cs.LG cs.AI stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

Scaling issues are mundane yet irritating for practitioners of reinforcement learning. Error scales vary across domains, tasks, and stages of learning; sometimes by many orders of magnitude. This can be detrimental to learning speed and stability, create interference between learning tasks, and necessitate substantial tuning. We revisit this topic for agents based on temporal-difference learning, sketch out some desiderata and investigate scenarios where simple fixes fall short. The mechanism we propose requires neither tuning, clipping, nor adaptation. We validate its effectiveness and robustness on the suite of Atari games. Our scaling method turns out to be particularly helpful at mitigating interference, when training a shared neural network on multiple targets that differ in reward scale or discounting.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 24 of its 43 references that are in the map. First posted on arXiv 11 May 2021.

Builds on

24 of 43 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Layer Normalizationcited by 5,579
  4. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  6. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  7. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  8. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  9. 2017Population Based Training of Neural Networkscited by 351
  10. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  11. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  12. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
12 more
  1. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  2. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  3. 2018Meta-Gradient Reinforcement Learningcited by 156
  4. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  5. 2018Generalization and Regularization in DQNcited by 117
  6. 2016Learning values across many orders of magnitudecited by 80
  7. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 78
  8. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
  9. 2017Deep Q-learning from Demonstrationscited by 27
  10. 2019Adapting Behaviour for Learning Progresscited by 11
  11. 2019General non-linear Bellman equationscited by 8
  12. 2021Adapting to Reward Progressivity via Spectral Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 1; 2024: 5; 2025: 4; 2026: 6.

22 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2024Variational Best-of-N Alignmentcited by 49
  3. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  4. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  5. 2022The Phenomenon of Policy Churncited by 28
  6. 2022Human-level Atari 200x fastercited by 28
  7. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  8. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 21
  9. 2021When should agents explore?cited by 20
  10. 2024Reward Centeringcited by 16
  11. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
  12. 2023BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximationscited by 13
10 more
  1. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  2. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  3. 2025Safe Reinforcement Learning for Real-World Engine Controlcited by 3
  4. 2022Backward Curriculum Reinforcement Learningcited by 0
  5. 2024OPTIMA: Optimized Policy for Intelligent Multi-Agent Systems Enables Coordination-Aware Autonomous Vehiclescited by 0
  6. 2026Incremental Residual Reinforcement Learning Toward Real-World Learning for Social Navigationcited by 0
  7. 2026Learning to Reason Efficiently with A* Post-Trainingcited by 0
  8. 2026Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learningcited by 0
  9. 2026Augmenting PID Control with Deep Reinforcement Learning: A Hybrid Approach to the Industrial Benchmarkcited by 0
  10. 2026Homomorphic Advantage Operator: Stabilizing Reinforcement Learning Under Fully Homomorphic Encryption Constraintscited by 0