mlsanityData to 14 Oct 2026

Why Batch Normalization Damage Federated Learning on Non-IID Data?

arXiv 2301.02982 ↗First posted 8 Jan 2023cs.LG cs.DCcited by 29

Not on the map. Why?

Abstract (arXiv)

As a promising distributed learning paradigm, federated learning (FL) involves training deep neural network (DNN) models at the network edge while protecting the privacy of the edge clients. To train a large-scale DNN model, batch normalization (BN) has been regarded as a simple and effective means to accelerate the training and improve the generalization capability. However, recent findings indicate that BN can significantly impair the performance of FL in the presence of non-i.i.d. data. While several FL algorithms have been proposed to address this issue, their performance still falls significantly when compared to the centralized scheme. Furthermore, none of them have provided a theoretical explanation of how the BN damages the FL convergence. In this paper, we present the first convergence analysis to show that under the non-i.i.d. data, the mismatch between the local and global statistical parameters in BN causes the gradient deviation between the local and global models, which, as a result, slows down and biases the FL convergence. In view of this, we develop a new FL algorithm that is tailored to BN, called FedTAN, which is capable of achieving robust FL performance under a variety of data distributions via iterative layer-wise parameter aggregation. Comprehensive experimental results demonstrate the superiority of the proposed FedTAN over existing baselines for training BN-based DNN models.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 8 Jan 2023.

Builds on

29 of 43 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2018Group Normalizationcited by 1,524
  6. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  9. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  10. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  11. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  12. 2018How Does Batch Normalization Help Optimization?cited by 357
17 more
  1. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  2. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  3. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  4. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  5. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  6. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  7. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  8. 2018Decorrelated Batch Normalizationcited by 83
  9. 2020Siloed Federated Learning for Multi-Centric Histopathology Datasetscited by 61
  10. 2020Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computingcited by 52
  11. 2020Design and Analysis of Uplink and Downlink Communications for Federated Learningcited by 48
  12. 2020Passport-aware Normalization for Deep Model Protectioncited by 43
  13. 2021Quantized Federated Learning under Transmission Delay and Outage Constraintscited by 37
  14. 2021Cost-Effective Federated Learning in Mobile Edge Networkscited by 25
  15. 2022Rethinking Normalization Methods in Federated Learningcited by 19
  16. 2020Batch Group Normalizationcited by 9
  17. 2022The Importance of (Exponentially More) Computing Powercited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 11; 2025: 10; 2026: 1.

29 papers.

Sort:

  1. 2023FedWon: Triumphing Multi-domain Federated Learning Without Normalizationcited by 11
  2. 2024Mobility Accelerates Learning: Convergence Analysis on Hierarchical Federated Learning in Vehicular Networkscited by 11
  3. 2024Training Machine Learning models at the Edge: A Surveycited by 11
  4. 2023Making Batch Normalization Great in Federated Deep Learningcited by 8
  5. 2024Fast-Convergent and Communication-Alleviated Heterogeneous Hierarchical Federated Learning in Autonomous Drivingcited by 8
  6. 2023Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergencecited by 5
  7. 2024Federated brain tumor segmentation: an extensive benchmarkcited by 5
  8. 2025A Thorough Assessment of the Non-IID Data Impact in Federated Learningcited by 5
  9. 2024Overcoming the Challenges of Batch Normalization in Federated Learningcited by 4
  10. 2024Towards Satellite Non-IID Imagery: A Spectral Clustering-Assisted Federated Learning Approachcited by 4
  11. 2025An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Datacited by 4
  12. 2024Federated and Transfer Learning for Cancer Detection Based on Image Analysiscited by 3
17 more
  1. 2025E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizingcited by 3
  2. 2023Understanding the Role of Layer Normalization in Label-Skewed Federated Learningcited by 2
  3. 2023Privacy-preserving Federated Primal-dual Learning for Non-convex and Non-smooth Problems with Model Sparsificationcited by 2
  4. 2025Communication-Efficient Federated Learning by Quantized Variance Reduction for Heterogeneous Wireless Edge Networkscited by 2
  5. 2025FedCGD: Collective Gradient Divergence Optimized Scheduling for Wireless Federated Learningcited by 2
  6. 2023Advocating for the Silent: Enhancing Federated Generalization for Non-Participating Clientscited by 1
  7. 2024FedStein: Enhancing Multi-Domain Federated Learning Through James-Stein Estimatorcited by 1
  8. 2025On the Out-of-Distribution Backdoor Attack for Federated Learningcited by 1
  9. 2023Optimizing Federated Learning for Medical Image Classification on Distributed Non-iid Datasets with Partial Labelscited by 0
  10. 2024Data Quality in Edge Machine Learning: A State-of-the-Art Surveycited by 0
  11. 2024FedRepOpt: Gradient Re-parametrized Optimizers in Federated Learningcited by 0
  12. 2024An Architecture Built for Federated Learning: Addressing Data Heterogeneity through Adaptive Normalization-Free Feature Recalibrationcited by 0
  13. 2025Robust Federated Learning in Unreliable Wireless Networks: A Client Selection Approachcited by 0
  14. 2025Stratify: Rethinking Federated Learning for Non-IID Data through Balanced Samplingcited by 0
  15. 2025Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learningcited by 0
  16. 2025Validation of Various Normalization Methods for Brain Tumor Segmentation: Can Federated Learning Overcome This Heterogeneity?cited by 0
  17. 2026Federated Learning for Feature Generalization with Convex Constraintscited by 0