mlsanityData to 14 Oct 2026

Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization

arXiv 2502.07523 ↗First posted 11 Feb 2025cs.LG cs.AIcited by 19

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning has achieved significant milestones, but sample efficiency remains a bottleneck for real-world applications. Recently, CrossQ has demonstrated state-of-the-art sample efficiency with a low update-to-data (UTD) ratio of 1. In this work, we explore CrossQ's scaling behavior with higher UTD ratios. We identify challenges in the training dynamics, which are emphasized by higher UTD ratios. To address these, we integrate weight normalization into the CrossQ framework, a solution that stabilizes training, has been shown to prevent potential loss of plasticity and keeps the effective learning rate constant. Our proposed approach reliably scales with increasing UTD ratios, achieving competitive performance across 25 challenging continuous control tasks on the DeepMind Control Suite and Myosuite benchmarks, notably the complex dog and humanoid environments. This work eliminates the need for drastic interventions, such as network resets, and offers a simple yet robust pathway for improving sample efficiency and scalability in model-free reinforcement learning.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 33 of its 47 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

33 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning13 Oct
  2. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL11 Oct
  3. 2024Normalization and effective learning rates in reinforcement learning1 Jul
  4. 2024Weight Clipping for Deep Continual and Reinforcement Learning1 Jul
  5. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learning25 Jun
  6. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May

Other references · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Layer Normalizationcited by 5,669
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2019Root Mean Square Layer Normalizationcited by 1,480
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  8. 2018DeepMind Control Suitecited by 956
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  11. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  12. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
15 more
  1. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  2. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  3. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  4. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  5. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  6. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  7. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  8. 2019When to use parametric models in reinforcement learning?cited by 118
  9. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  10. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  11. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  12. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  13. 2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80
  14. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  15. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 12.

19 papers.

Sort:

By area:

  1. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 38
  2. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  3. 2025Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotioncited by 8
  4. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  5. 2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4
  6. 2025Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learningcited by 2
  7. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  8. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1
  9. 2026XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policiescited by 1
  10. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  11. 2026Behavior-Consistent Deep Reinforcement Learningcited by 1
  12. 2025An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithmscited by 0
7 more
  1. 2026Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learningcited by 0
  2. 2026Extending Differential Temporal Difference Methods for Episodic Problemscited by 0
  3. 2026Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewardscited by 0
  4. 2026Momba: Network Modernization Improves Multi-Objective Reinforcement Learningcited by 0
  5. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0
  6. 2026On BatchNorm Forward Modes in Value-Based Reinforcement Learningcited by 0
  7. 2026Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Criticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling CrossQ with Weight Normalization
  2. 2025SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learning
  3. 2025Compute-Optimal Scaling for Value-Based Deep RL
  4. 2023Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting
  5. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
  6. 2026Scalable Reinforcement Learning via Adaptive Batch Scaling
  7. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
  8. 2026Deep Weighted Bellman Residual Minimization for Q^* Estimation