# EvalNorm: Estimating Batch Normalization Statistics for Evaluation

arXiv: https://arxiv.org/abs/1904.06031 · HTML: https://mlsanity.com/paper/1904.06031/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2019.

## Built on by (most cited first, 20 of 20)

- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization](https://mlsanity.com/paper/2112.00463/) (2021)
- [TTN: A Domain-Shift Aware Batch Normalization in Test-Time Adaptation](https://mlsanity.com/paper/2302.05155/) (2023)
- [An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions](https://mlsanity.com/paper/2204.08817/) (2022)
- [Understanding AdamW through Proximal Methods and Scale-Freeness](https://mlsanity.com/paper/2202.00089/) (2022)
- [Rethinking "Batch" in BatchNorm](https://mlsanity.com/paper/2105.07576/) (2021)
- [On the Pitfalls of Batch Normalization for End-to-End Video Learning: A Study on Surgical Workflow Analysis](https://mlsanity.com/paper/2203.07976/) (2022)
- [Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning](https://mlsanity.com/paper/2101.07525/) (2021)
- [Understanding the Failure of Batch Normalization for Transformers in NLP](https://mlsanity.com/paper/2210.05153/) (2022)
- [Uncovering Adversarial Risks of Test-Time Adaptation](https://mlsanity.com/paper/2301.12576/) (2023)
- [MedBN: Robust Test-Time Adaptation against Malicious Test Samples](https://mlsanity.com/paper/2403.19326/) (2024)
- [Delving into the Estimation Shift of Batch Normalization in a Network](https://mlsanity.com/paper/2203.10778/) (2022)
- [Decoupled Prototype Learning for Reliable Test-Time Adaptation](https://mlsanity.com/paper/2401.08703/) (2024)
- [Diagnosing Batch Normalization in Class Incremental Learning](https://mlsanity.com/paper/2202.08025/) (2022)
- [FedConv: Enhancing Convolutional Neural Networks for Handling Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2310.04412/) (2023)
- [Ghost Noise for Regularizing Deep Neural Networks](https://mlsanity.com/paper/2305.17205/) (2023)
- [On Sensitivity and Robustness of Normalization Schemes to Input Distribution Shifts in Automatic MR Image Diagnosis](https://mlsanity.com/paper/2306.13276/) (2023)
- [L2BN: Enhancing Batch Normalization by Equalizing the L2 Norms of Features](https://mlsanity.com/paper/2207.02625/) (2022)
- [Meta-TTT: A Meta-learning Minimax Framework For Test-Time Training](https://mlsanity.com/paper/2410.01709/) (2024)
- [Spooky Action at a Distance: Normalization Layers Enable Side-Channel Spatial Communication](https://mlsanity.com/paper/2507.04709/) (2025)

## Abstract (arXiv)

Batch normalization (BN) has been very effective for deep learning and is widely used. However, when training with small minibatches, models using BN exhibit a significant degradation in performance. In this paper we study this peculiar behavior of BN to gain a better understanding of the problem, and identify a cause. We propose 'EvalNorm' to address the issue by estimating corrected normalization statistics to use for BN during evaluation. EvalNorm supports online estimation of the corrected statistics while the model is being trained, and does not affect the training scheme of the model. As a result, EvalNorm can also be used with existing pre-trained models allowing them to benefit from our method. EvalNorm yields large gains for models trained with smaller batches. Our experiments show that EvalNorm performs 6.18% (absolute) better than vanilla BN for a batchsize of 2 on ImageNet validation set and from 1.5 to 7.0 points (absolute) gain on the COCO object detection benchmark across a variety of setups.