mlsanityData to 14 Oct 2026

An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms

arXiv 2509.23750 ↗First posted 28 Sep 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Batch Normalization (BN) has played a pivotal role in the success of deep learning by improving training stability, mitigating overfitting, and enabling more effective optimization. However, its adoption in deep reinforcement learning (DRL) has been limited due to the inherent non-i.i.d. nature of data and the dynamically shifting distributions induced by the agent's learning process. In this paper, we argue that, despite these challenges, BN retains unique advantages in DRL settings, particularly through its stochasticity and its ability to ease training. When applied appropriately, BN can adapt to evolving data distributions and enhance both convergence speed and final performance. To this end, we conduct a comprehensive empirical study on the use of BN in off-policy actor-critic algorithms, systematically analyzing how different training and evaluation modes impact performance. We further identify failure modes that lead to instability or divergence, analyze their underlying causes, and propose the Mode-Aware Batch Normalization (MA-BN) method with practical actionable recommendations for robust BN integration in DRL pipelines. We also empirically validate that, in RL settings, MA-BN accelerates and stabilizes training, broadens the effective learning rate range, enhances exploration, and reduces overall optimization difficulty. Our code is available at: https://github.com/monster476/ma-bn.git.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 30 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

23 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization11 Feb
  2. 2024Normalization and effective learning rates in reinforcement learning1 Jul

Other references · 21

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  7. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 431
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  9. 2018How Does Batch Normalization Help Optimization?cited by 357
  10. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  11. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  12. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
9 more
  1. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 128
  2. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  3. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  4. 2019Iterative Normalization: Beyond Standardization towards Efficient Whiteningcited by 92
  5. 2018Bayesian Uncertainty Estimation for Batch Normalized Deep Networkscited by 70
  6. 2023Heterogeneous-Agent Reinforcement Learningcited by 69
  7. 2020An Investigation into the Stochasticity of Batch Whiteningcited by 14
  8. 2022Delving into the Estimation Shift of Batch Normalization in a Networkcited by 7
  9. 2018Stochastic Normalizations as Bayesian Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
  2. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
  3. 2025CaRe-BN: Precise Moving Statistics for Stabilizing Spiking Neural Networks in Reinforcement Learning
  4. 2020Critic Regularized Regression
  5. 2026NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning
  6. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
  7. 2018Off-Policy Deep Reinforcement Learning without Exploration
  8. 2026ReBRAC-v2: The Return of the King