mlsanityData to 7 Oct 2026

Making Batch Normalization Great in Federated Deep Learning

arXiv 2303.06530 ↗First posted 12 Mar 2023cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Batch Normalization (BN) is widely used in {centralized} deep learning to improve convergence and generalization. However, in {federated} learning (FL) with decentralized data, prior work has observed that training with BN could hinder performance and suggested replacing it with Group Normalization (GN). In this paper, we revisit this substitution by expanding the empirical study conducted in prior work. Surprisingly, we find that BN outperforms GN in many FL settings. The exceptions are high-frequency communication and extreme non-IID regimes. We reinvestigate factors that are believed to cause this problem, including the mismatch of BN statistics across clients and the deviation of gradients during local training. We empirically identify a simple practice that could reduce the impacts of these factors while maintaining the strength of BN. Our approach, which we named FIXBN, is fairly easy to implement, without any additional training or communication costs, and performs favorably across a wide range of FL settings. We hope that our study could serve as a valuable reference for future practical usage and theoretical analysis in FL.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 64 of its 90 references that are in the map. First posted on arXiv 12 Mar 2023.

Builds on

64 of 90 references.

References · 64

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Layer Normalizationcited by 5,579
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  8. 2019Advances and Open Problems in Federated Learningcited by 3,021
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  10. 2018Group Normalizationcited by 1,514
  11. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  12. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
38 more
  1. 2018Federated Learning with Non-IID Datacited by 1,099
  2. 2020Adaptive Federated Optimizationcited by 1,036
  3. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  4. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  5. 2017Federated Multi-Task Learningcited by 669
  6. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  7. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  8. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  9. 2020Federated Learning with Matched Averagingcited by 564
  10. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  11. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  12. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  13. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  14. 2018How Does Batch Normalization Help Optimization?cited by 355
  15. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  16. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  17. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  18. 2016Revisiting Batch Normalization For Practical Domain Adaptationcited by 285
  19. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  20. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  21. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  22. 2018Understanding Batch Normalizationcited by 157
  23. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  24. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 150
  25. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  26. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 140
  27. 2020Distilled One-Shot Federated Learningcited by 108
  28. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 91
  29. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  30. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  31. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  32. 2021On Large-Cohort Training for Federated Learningcited by 73
  33. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  34. 2020FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tierscited by 64
  35. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  36. 2020Siloed Federated Learning for Multi-Centric Histopathology Datasetscited by 61
  37. 2019A Mean Field Theory of Batch Normalizationcited by 58
  38. 2021Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharingcited by 58

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 3; 2025: 2.

8 papers.

  1. 2023FedWon: Triumphing Multi-domain Federated Learning Without Normalizationcited by 11
  2. 2023Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Datacited by 7
  3. 2024Backpropagation-Free Multi-modal On-Device Model Adaptation via Cloud-Device Collaborationcited by 5
  4. 2024Overcoming the Challenges of Batch Normalization in Federated Learningcited by 3
  5. 2023ARIA: On the Interaction Between Architectures, Initialization and Aggregation Methods for Federated Visual Classificationcited by 1
  6. 2025Patient-Specific Autoregressive Models for Organ Motion Prediction in Radiotherapycited by 1
  7. 2024An Architecture Built for Federated Learning: Addressing Data Heterogeneity through Adaptive Normalization-Free Feature Recalibrationcited by 0
  8. 2025Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learningcited by 0