mlsanityData to 14 Oct 2026

Overcoming the Challenges of Batch Normalization in Federated Learning

arXiv 2405.14670 ↗First posted 23 May 2024cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Batch normalization has proven to be a very beneficial mechanism to accelerate the training and improve the accuracy of deep neural networks in centralized environments. Yet, the scheme faces significant challenges in federated learning, especially under high data heterogeneity. Essentially, the main challenges arise from external covariate shifts and inconsistent statistics across clients. We introduce in this paper Federated BatchNorm (FBN), a novel scheme that restores the benefits of batch normalization in federated learning. Essentially, FBN ensures that the batch normalization during training is consistent with what would be achieved in a centralized execution, hence preserving the distribution of the data, and providing running statistics that accurately approximate the global statistics. FBN thereby reduces the external covariate shift and matches the evaluation performance of the centralized setting. We also show that, with a slight increase in complexity, we can robustify FBN to mitigate erroneous statistics and potentially adversarial attacks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 25 of its 30 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

25 of 30 references.

References · 25

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Layer Normalizationcited by 5,669
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2018Group Normalizationcited by 1,524
  7. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  8. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
  9. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  10. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  11. 2018How Does Batch Normalization Help Optimization?cited by 357
  12. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
13 more
  1. 2019A Little Is Enough: Circumventing Defenses For Distributed Learningcited by 288
  2. 2019Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulationcited by 141
  3. 2020Byzantine-Robust Learning on Heterogeneous Datasets via Bucketingcited by 128
  4. 2022NVIDIA FLARE: Federated Learning from Simulation to Real-Worldcited by 80
  5. 2023Fixing by Mixing: A Recipe for Optimal Byzantine ML under Heterogeneitycited by 65
  6. 2022Byzantine Machine Learning Made Easy by Resilient Averaging of Momentumscited by 63
  7. 2020Byzantine-Resilient Non-Convex Stochastic Gradient Descentcited by 57
  8. 2023Why Batch Normalization Damage Federated Learning on Non-IID Data?cited by 29
  9. 2022Rethinking Normalization Methods in Federated Learningcited by 19
  10. 2018Universal Multi-Party Poisoning Attackscited by 17
  11. 2022Robust Collaborative Learning with Linear Gradient Overheadcited by 12
  12. 2023FedWon: Triumphing Multi-domain Federated Learning Without Normalizationcited by 11
  13. 2023Making Batch Normalization Great in Federated Deep Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2026BlackCATT: Black-box Collusion Aware Traitor Tracing in Federated Learningcited by 1
  2. 2024An Architecture Built for Federated Learning: Addressing Data Heterogeneity through Adaptive Normalization-Free Feature Recalibrationcited by 0
  3. 2025Federated Learning for Predicting Mild Cognitive Impairment to Dementia Conversioncited by 0
  4. 2025Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learningcited by 0