mlsanityData to 14 Oct 2026

Scaling CrossQ with Weight Normalization

arXiv 2506.03758 ↗First posted 4 Jun 2025cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning has achieved significant milestones, but sample efficiency remains a bottleneck for real-world applications. Recently, CrossQ has demonstrated state-of-the-art sample efficiency with a low update-to-data (UTD) ratio of 1. In this work, we explore CrossQ's scaling behavior with higher UTD ratios. We identify challenges in the training dynamics which are emphasized by higher UTDs, particularly Q-bias explosion and the growing magnitude of critic network weights. To address this, we integrate weight normalization into the CrossQ framework, a solution that stabilizes training, prevents potential loss of plasticity and keeps the effective learning rate constant. Our proposed approach reliably scales with increasing UTD ratios, achieving competitive or superior performance across a range of challenging tasks on the DeepMind control benchmark, notably the complex dog and humanoid environments. This work eliminates the need for drastic interventions, such as network resets, and offers a robust pathway for improving sample efficiency and scalability in model-free reinforcement learning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 18 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

12 of 18 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Normalization and effective learning rates in reinforcement learning1 Jul
  2. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May

Other references · 10

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  2. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  3. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  4. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  5. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  6. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  7. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  8. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  9. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  10. 2020SAMBA: Safe Model-Based & Active Reinforcement Learningcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization
  2. 2025Compute-Optimal Scaling for Value-Based Deep RL
  3. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
  4. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergence
  5. 2025SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learning
  6. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning
  7. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
  8. 2025The Art of Scaling Reinforcement Learning Compute for LLMs