# Overcoming the Challenges of Batch Normalization in Federated Learning

arXiv: https://arxiv.org/abs/2405.14670 · HTML: https://mlsanity.com/paper/2405.14670/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 30 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 25 of 25)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates](https://mlsanity.com/paper/1803.01498/) (2018)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [A Little Is Enough: Circumventing Defenses For Distributed Learning](https://mlsanity.com/paper/1902.06156/) (2019)
- [Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation](https://mlsanity.com/paper/1903.03936/) (2019)
- [Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing](https://mlsanity.com/paper/2006.09365/) (2020)
- [NVIDIA FLARE: Federated Learning from Simulation to Real-World](https://mlsanity.com/paper/2210.13291/) (2022)
- [Fixing by Mixing: A Recipe for Optimal Byzantine ML under Heterogeneity](https://mlsanity.com/paper/2302.01772/) (2023)
- [Byzantine Machine Learning Made Easy by Resilient Averaging of Momentums](https://mlsanity.com/paper/2205.12173/) (2022)
- [Byzantine-Resilient Non-Convex Stochastic Gradient Descent](https://mlsanity.com/paper/2012.14368/) (2020)
- [Why Batch Normalization Damage Federated Learning on Non-IID Data?](https://mlsanity.com/paper/2301.02982/) (2023)
- [Rethinking Normalization Methods in Federated Learning](https://mlsanity.com/paper/2210.03277/) (2022)
- [Universal Multi-Party Poisoning Attacks](https://mlsanity.com/paper/1809.03474/) (2018)
- [Robust Collaborative Learning with Linear Gradient Overhead](https://mlsanity.com/paper/2209.10931/) (2022)
- [FedWon: Triumphing Multi-domain Federated Learning Without Normalization](https://mlsanity.com/paper/2306.05879/) (2023)
- [Making Batch Normalization Great in Federated Deep Learning](https://mlsanity.com/paper/2303.06530/) (2023)

## Built on by (most cited first, 4 of 4)

- [BlackCATT: Black-box Collusion Aware Traitor Tracing in Federated Learning](https://mlsanity.com/paper/2602.12138/) (2026)
- [An Architecture Built for Federated Learning: Addressing Data Heterogeneity through Adaptive Normalization-Free Feature Recalibration](https://mlsanity.com/paper/2410.02006/) (2024)
- [Federated Learning for Predicting Mild Cognitive Impairment to Dementia Conversion](https://mlsanity.com/paper/2503.03489/) (2025)
- [Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning](https://mlsanity.com/paper/2505.21877/) (2025)

## Abstract (arXiv)

Batch normalization has proven to be a very beneficial mechanism to accelerate the training and improve the accuracy of deep neural networks in centralized environments. Yet, the scheme faces significant challenges in federated learning, especially under high data heterogeneity. Essentially, the main challenges arise from external covariate shifts and inconsistent statistics across clients. We introduce in this paper Federated BatchNorm (FBN), a novel scheme that restores the benefits of batch normalization in federated learning. Essentially, FBN ensures that the batch normalization during training is consistent with what would be achieved in a centralized execution, hence preserving the distribution of the data, and providing running statistics that accurately approximate the global statistics. FBN thereby reduces the external covariate shift and matches the evaluation performance of the centralized setting. We also show that, with a slight increase in complexity, we can robustify FBN to mitigate erroneous statistics and potentially adversarial attacks.