mlsanityData to 7 Oct 2026

On BatchNorm Forward Modes in Value-Based Reinforcement Learning

arXiv 2609.06421 ↗First posted 6 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Batch normalization (BN) substantially improves sample efficiency in continuous-control actor-critic methods such as CrossQ, yet recent studies report performance degradation in discrete-action value learning on Atari. These failures are surprising because discrete Q-networks lack the action-input distribution mismatch identified by CrossQ. We show for target-based C51 and target-free PQN that the simple choice between running and batch statistics at specific forward passes can reverse this degradation. In C51, switching the BN bootstrap forward to batch-statistic mode significantly improves performance over unnormalized and LayerNorm baselines and scales stably with update-to-data ratios up to 12. In PQN, using batch-statistics for both action selection and bootstrapping recovers performance from the failing running-statistic configuration. Across 26 Atari games at 400M frames, this configuration achieves a higher final aggregate score than PQN with LayerNorm. Our results show that carefully configured BN can substantially improve discrete-action value learning, and that its forward protocols are an essential part of the algorithm specification.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 23 references that are in the map. First posted on arXiv 6 Sep 2026.

Builds on

20 of 23 references.

References · 20

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  5. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  6. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  7. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  8. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  9. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  10. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
  11. 2024Simplifying Deep Temporal Difference Learningcited by 71
  12. 2018Improving Regression Performance with Distributional Lossescited by 66
8 more
  1. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  2. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  3. 2022Continual Normalization: Rethinking Batch Normalization for Online Continual Learningcited by 40
  4. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  5. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  6. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  7. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  8. 2026XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policiescited by 1

Built on by

Not cited yet.