# Why Batch Normalization Damage Federated Learning on Non-IID Data?

arXiv: https://arxiv.org/abs/2301.02982 · HTML: https://mlsanity.com/paper/2301.02982/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 8 Jan 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization](https://mlsanity.com/paper/1909.13014/) (2019)
- [HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients](https://mlsanity.com/paper/2010.01264/) (2020)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning](https://mlsanity.com/paper/1807.06629/) (2018)
- [Model Pruning Enables Efficient Federated Learning on Edge Devices](https://mlsanity.com/paper/1909.12326/) (2019)
- [Evaluating Gradient Inversion Attacks and Defenses in Federated Learning](https://mlsanity.com/paper/2112.00059/) (2021)
- [Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2106.06047/) (2021)
- [Decorrelated Batch Normalization](https://mlsanity.com/paper/1804.08450/) (2018)
- [Siloed Federated Learning for Multi-Centric Histopathology Datasets](https://mlsanity.com/paper/2008.07424/) (2020)
- [Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing](https://mlsanity.com/paper/2007.09236/) (2020)
- [Design and Analysis of Uplink and Downlink Communications for Federated Learning](https://mlsanity.com/paper/2012.04057/) (2020)
- [Passport-aware Normalization for Deep Model Protection](https://mlsanity.com/paper/2010.15824/) (2020)
- [Quantized Federated Learning under Transmission Delay and Outage Constraints](https://mlsanity.com/paper/2106.09397/) (2021)
- [Cost-Effective Federated Learning in Mobile Edge Networks](https://mlsanity.com/paper/2109.05411/) (2021)
- [Rethinking Normalization Methods in Federated Learning](https://mlsanity.com/paper/2210.03277/) (2022)
- [Batch Group Normalization](https://mlsanity.com/paper/2012.02782/) (2020)
- [The Importance of (Exponentially More) Computing Power](https://mlsanity.com/paper/2206.14007/) (2022)

## Built on by (most cited first, 29 of 29)

- [FedWon: Triumphing Multi-domain Federated Learning Without Normalization](https://mlsanity.com/paper/2306.05879/) (2023)
- [Mobility Accelerates Learning: Convergence Analysis on Hierarchical Federated Learning in Vehicular Networks](https://mlsanity.com/paper/2401.09656/) (2024)
- [Training Machine Learning models at the Edge: A Survey](https://mlsanity.com/paper/2403.02619/) (2024)
- [Making Batch Normalization Great in Federated Deep Learning](https://mlsanity.com/paper/2303.06530/) (2023)
- [Fast-Convergent and Communication-Alleviated Heterogeneous Hierarchical Federated Learning in Autonomous Driving](https://mlsanity.com/paper/2409.19560/) (2024)
- [Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergence](https://mlsanity.com/paper/2302.13562/) (2023)
- [Federated brain tumor segmentation: an extensive benchmark](https://mlsanity.com/paper/2410.17265/) (2024)
- [A Thorough Assessment of the Non-IID Data Impact in Federated Learning](https://mlsanity.com/paper/2503.17070/) (2025)
- [Overcoming the Challenges of Batch Normalization in Federated Learning](https://mlsanity.com/paper/2405.14670/) (2024)
- [Towards Satellite Non-IID Imagery: A Spectral Clustering-Assisted Federated Learning Approach](https://mlsanity.com/paper/2410.13602/) (2024)
- [An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data](https://mlsanity.com/paper/2509.05213/) (2025)
- [Federated and Transfer Learning for Cancer Detection Based on Image Analysis](https://mlsanity.com/paper/2405.20126/) (2024)
- [E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing](https://mlsanity.com/paper/2502.03092/) (2025)
- [Understanding the Role of Layer Normalization in Label-Skewed Federated Learning](https://mlsanity.com/paper/2308.09565/) (2023)
- [Privacy-preserving Federated Primal-dual Learning for Non-convex and Non-smooth Problems with Model Sparsification](https://mlsanity.com/paper/2310.19558/) (2023)
- [Communication-Efficient Federated Learning by Quantized Variance Reduction for Heterogeneous Wireless Edge Networks](https://mlsanity.com/paper/2501.11267/) (2025)
- [FedCGD: Collective Gradient Divergence Optimized Scheduling for Wireless Federated Learning](https://mlsanity.com/paper/2506.07581/) (2025)
- [Advocating for the Silent: Enhancing Federated Generalization for Non-Participating Clients](https://mlsanity.com/paper/2310.07171/) (2023)
- [FedStein: Enhancing Multi-Domain Federated Learning Through James-Stein Estimator](https://mlsanity.com/paper/2410.03499/) (2024)
- [On the Out-of-Distribution Backdoor Attack for Federated Learning](https://mlsanity.com/paper/2509.13219/) (2025)
- [Optimizing Federated Learning for Medical Image Classification on Distributed Non-iid Datasets with Partial Labels](https://mlsanity.com/paper/2303.06180/) (2023)
- [Data Quality in Edge Machine Learning: A State-of-the-Art Survey](https://mlsanity.com/paper/2406.02600/) (2024)
- [FedRepOpt: Gradient Re-parametrized Optimizers in Federated Learning](https://mlsanity.com/paper/2409.15898/) (2024)
- [An Architecture Built for Federated Learning: Addressing Data Heterogeneity through Adaptive Normalization-Free Feature Recalibration](https://mlsanity.com/paper/2410.02006/) (2024)
- [Robust Federated Learning in Unreliable Wireless Networks: A Client Selection Approach](https://mlsanity.com/paper/2502.17260/) (2025)
- [Stratify: Rethinking Federated Learning for Non-IID Data through Balanced Sampling](https://mlsanity.com/paper/2504.13462/) (2025)
- [Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning](https://mlsanity.com/paper/2505.21877/) (2025)
- [Validation of Various Normalization Methods for Brain Tumor Segmentation: Can Federated Learning Overcome This Heterogeneity?](https://mlsanity.com/paper/2510.07126/) (2025)
- [Federated Learning for Feature Generalization with Convex Constraints](https://mlsanity.com/paper/2606.14416/) (2026)

## Abstract (arXiv)

As a promising distributed learning paradigm, federated learning (FL) involves training deep neural network (DNN) models at the network edge while protecting the privacy of the edge clients. To train a large-scale DNN model, batch normalization (BN) has been regarded as a simple and effective means to accelerate the training and improve the generalization capability. However, recent findings indicate that BN can significantly impair the performance of FL in the presence of non-i.i.d. data. While several FL algorithms have been proposed to address this issue, their performance still falls significantly when compared to the centralized scheme. Furthermore, none of them have provided a theoretical explanation of how the BN damages the FL convergence. In this paper, we present the first convergence analysis to show that under the non-i.i.d. data, the mismatch between the local and global statistical parameters in BN causes the gradient deviation between the local and global models, which, as a result, slows down and biases the FL convergence. In view of this, we develop a new FL algorithm that is tailored to BN, called FedTAN, which is capable of achieving robust FL performance under a variety of data distributions via iterative layer-wise parameter aggregation. Comprehensive experimental results demonstrate the superiority of the proposed FedTAN over existing baselines for training BN-based DNN models.