# CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity

arXiv: https://arxiv.org/abs/1902.05605 · HTML: https://mlsanity.com/paper/1902.05605/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2019.

## Built on by (most cited first, 30 of 95)

- [Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control](https://mlsanity.com/paper/2405.16158/) (2024)
- [Simplifying Deep Temporal Difference Learning](https://mlsanity.com/paper/2407.04811/) (2024)
- [SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning](https://mlsanity.com/paper/2410.09754/) (2024)
- [DIME:Diffusion-Based Maximum Entropy Reinforcement Learning](https://mlsanity.com/paper/2502.02316/) (2025)
- [Latent Action Learning Requires Supervision in the Presence of Distractors](https://mlsanity.com/paper/2502.00379/) (2025)
- [Hyperspherical Normalization for Scalable Deep Reinforcement Learning](https://mlsanity.com/paper/2502.15280/) (2025)
- [Maximum Entropy Reinforcement Learning with Diffusion Policy](https://mlsanity.com/paper/2502.11612/) (2025)
- [Plasticity Loss in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2411.04832/) (2024)
- [Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners](https://mlsanity.com/paper/2505.23150/) (2025)
- [floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL](https://mlsanity.com/paper/2509.06863/) (2025)
- [Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation](https://mlsanity.com/paper/2412.12089/) (2024)
- [Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization](https://mlsanity.com/paper/2502.07523/) (2025)
- [Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning](https://mlsanity.com/paper/2405.14956/) (2024)
- [Dissecting Deep RL with High Update Ratios: Combatting Value Divergence](https://mlsanity.com/paper/2403.05996/) (2024)
- [XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning](https://mlsanity.com/paper/2509.25174/) (2025)
- [Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns](https://mlsanity.com/paper/2503.03660/) (2025)
- [BlendRL: A Framework for Merging Symbolic and Neural Policy Learning](https://mlsanity.com/paper/2410.11689/) (2024)
- [A System for Morphology-Task Generalization via Unified Representation and Behavior Distillation](https://mlsanity.com/paper/2211.14296/) (2022)
- [Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17204/) (2025)
- [FRASA: An End-to-End Reinforcement Learning Agent for Fall Recovery and Stand Up of Humanoid Robots](https://mlsanity.com/paper/2410.08655/) (2024)
- [Convergent and Efficient Deep Q Network Algorithm](https://mlsanity.com/paper/2106.15419/) (2021)
- [Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotion](https://mlsanity.com/paper/2503.08375/) (2025)
- [Hadamax Encoding: Elevating Performance in Model-Free Atari](https://mlsanity.com/paper/2505.15345/) (2025)
- [Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning](https://mlsanity.com/paper/2111.12673/) (2021)
- [On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economics](https://mlsanity.com/paper/2310.19527/) (2023)
- [Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning](https://mlsanity.com/paper/2403.02107/) (2024)
- [Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learning](https://mlsanity.com/paper/2503.01437/) (2025)
- [Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow](https://mlsanity.com/paper/2601.19707/) (2026)
- [FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control](https://mlsanity.com/paper/2604.04539/) (2026)
- [Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning](https://mlsanity.com/paper/2506.04398/) (2025)

## Abstract (arXiv)

Sample efficiency is a crucial problem in deep reinforcement learning. Recent algorithms, such as REDQ and DroQ, found a way to improve the sample efficiency by increasing the update-to-data (UTD) ratio to 20 gradient update steps on the critic per environment sample. However, this comes at the expense of a greatly increased computational cost. To reduce this computational burden, we introduce CrossQ: A lightweight algorithm for continuous control tasks that makes careful use of Batch Normalization and removes target networks to surpass the current state-of-the-art in sample efficiency while maintaining a low UTD ratio of 1. Notably, CrossQ does not rely on advanced bias-reduction schemes used in current methods. CrossQ's contributions are threefold: (1) it matches or surpasses current state-of-the-art methods in terms of sample efficiency, (2) it substantially reduces the computational cost compared to REDQ and DroQ, (3) it is easy to implement, requiring just a few lines of code on top of SAC.