# Batch Normalization is a Cause of Adversarial Vulnerability

arXiv: https://arxiv.org/abs/1905.02161 · HTML: https://mlsanity.com/paper/1905.02161/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2019.

## Built on by (most cited first, 29 of 29)

- [The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization](https://mlsanity.com/paper/2112.00463/) (2021)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?](https://mlsanity.com/paper/2111.01124/) (2021)
- [Improving Adversarial Robustness via Channel-wise Activation Suppressing](https://mlsanity.com/paper/2103.08307/) (2021)
- [Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning](https://mlsanity.com/paper/2106.05956/) (2021)
- [Removing Batch Normalization Boosts Adversarial Training](https://mlsanity.com/paper/2207.01156/) (2022)
- [Shift Invariance Can Reduce Adversarial Robustness](https://mlsanity.com/paper/2103.02695/) (2021)
- [Feature Separation and Recalibration for Adversarial Robustness](https://mlsanity.com/paper/2303.13846/) (2023)
- [Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence](https://mlsanity.com/paper/2106.03743/) (2021)
- [What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space](https://mlsanity.com/paper/2101.06898/) (2021)
- [Domain-Specific Block Selection and Paired-View Pseudo-Labeling for Online Test-Time Adaptation](https://mlsanity.com/paper/2404.10966/) (2024)
- [Spectrum Extraction and Clipping for Implicitly Linear Layers](https://mlsanity.com/paper/2402.16017/) (2024)
- [Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Accelerators](https://mlsanity.com/paper/2104.08323/) (2021)
- [Bridging the Gap Between Adversarial Robustness and Optimization Bias](https://mlsanity.com/paper/2102.08868/) (2021)
- [Batch Normalization Preconditioning for Neural Network Training](https://mlsanity.com/paper/2108.01110/) (2021)
- [Fairness Increases Adversarial Vulnerability](https://mlsanity.com/paper/2211.11835/) (2022)
- [On Fragile Features and Batch Normalization in Adversarial Training](https://mlsanity.com/paper/2204.12393/) (2022)
- [Clustering Effect of (Linearized) Adversarial Robust Models](https://mlsanity.com/paper/2111.12922/) (2021)
- [Batch Normalization Is Blind to the First and Second Derivatives of the Loss](https://mlsanity.com/paper/2205.15146/) (2022)
- [Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting](https://mlsanity.com/paper/2410.16150/) (2024)
- [CausAdv: A Causal-based Framework for Detecting Adversarial Examples](https://mlsanity.com/paper/2411.00839/) (2024)
- [Test-Time Defense Against Adversarial Attacks via Stochastic Resonance of Latent Ensembles](https://mlsanity.com/paper/2510.03224/) (2025)
- [Investigating and unmasking feature-level vulnerabilities of CNNs to adversarial perturbations](https://mlsanity.com/paper/2405.20672/) (2024)
- [Batch Normalization Amplifies Memorization and Privacy Risks](https://mlsanity.com/paper/2605.24420/) (2026)
- [Dealing with Adversarial Player Strategies in the Neural Network Game iNNk through Ensemble Learning](https://mlsanity.com/paper/2107.02052/) (2021)
- [Counterbalancing Teacher: Regularizing Batch Normalized Models for Robustness](https://mlsanity.com/paper/2207.01548/) (2022)
- [Pitfalls of Conditional Batch Normalization for Contextual Multi-Modal Learning](https://mlsanity.com/paper/2211.15071/) (2022)
- [SOLAR: Switchable Output Layer for Accuracy and Robustness in Once-for-All Training](https://mlsanity.com/paper/2509.16833/) (2025)
- [Explaining Machine Learning and Memorization with Statistical Mechanics](https://mlsanity.com/paper/2606.31110/) (2026)

## Abstract (arXiv)

Batch normalization (batch norm) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, it also reduces robustness to small adversarial input perturbations and noise by double-digit percentages, as we show on five standard datasets. Furthermore, substituting weight decay for batch norm is sufficient to nullify the relationship between adversarial vulnerability and the input dimension. Our work is consistent with a mean-field analysis that found that batch norm causes exploding gradients.