mlsanityData to 7 Oct 2026

Rethinking Normalization Methods in Federated Learning

arXiv 2210.03277 ↗First posted 7 Oct 2022cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a popular distributed learning framework that can reduce privacy risks by not explicitly sharing private data. In this work, we explicitly uncover external covariate shift problem in FL, which is caused by the independent local training processes on different devices. We demonstrate that external covariate shifts will lead to the obliteration of some devices' contributions to the global model. Further, we show that normalization layers are indispensable in FL since their inherited properties can alleviate the problem of obliterating some devices' contributions. However, recent works have shown that batch normalization, which is one of the standard components in many deep neural networks, will incur accuracy drop of the global model in FL. The essential reason for the failure of batch normalization in FL is poorly studied. We unveil that external covariate shift is the key reason why batch normalization is ineffective in FL. We also show that layer normalization is a better choice in FL which can mitigate the external covariate shift and improve the performance of the global model. We conduct experiments on CIFAR10 under non-IID settings. The results demonstrate that models with layer normalization converge fastest and achieve the best or comparable accuracy for three different model architectures.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 15 of its 18 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

15 of 18 references.

References · 15

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Layer Normalizationcited by 5,579
  4. 2018Group Normalizationcited by 1,514
  5. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  6. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  7. 2018How Does Batch Normalization Help Optimization?cited by 355
  8. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  9. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  10. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  11. 2018On the importance of single directions for generalizationcited by 99
  12. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
3 more
  1. 2018Towards Understanding Regularization in Batch Normalizationcited by 56
  2. 2020New Interpretations of Normalization Methods in Deep Learningcited by 8
  3. 2015Data-Dependent Path Normalization in Neural Networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 5; 2025: 5; 2026: 1.

19 papers.

Sort:

By area:

  1. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  2. 2023Why Batch Normalization Damage Federated Learning on Non-IID Data?cited by 29
  3. 2023FedWon: Triumphing Multi-domain Federated Learning Without Normalizationcited by 11
  4. 2023Making Batch Normalization Great in Federated Deep Learningcited by 8
  5. 2024Overcoming the Challenges of Batch Normalization in Federated Learningcited by 3
  6. 2024ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Modelscited by 3
  7. 2024MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learningcited by 3
  8. 2023Understanding the Role of Layer Normalization in Label-Skewed Federated Learningcited by 2
  9. 2023FedConv: Enhancing Convolutional Neural Networks for Handling Data Heterogeneity in Federated Learningcited by 2
  10. 2023Point Transformer with Federated Learning for Predicting Breast Cancer HER2 Status from Hematoxylin and Eosin-Stained Whole Slide Imagescited by 1
  11. 2025MyDigiTwin: A Privacy-Preserving Framework for Personalized Cardiovascular Risk Prediction and Scenario Explorationcited by 1
  12. 2025Hear No Evil: Detecting Gradient Leakage by Malicious Servers in Federated Learningcited by 1
7 more
  1. 2023Feature Norm Regularized Federated Learning: Transforming Skewed Distributions into Global Insightscited by 0
  2. 2024StatAvg: Mitigating Data Heterogeneity in Federated Learning for Intrusion Detection Systemscited by 0
  3. 2024An Architecture Built for Federated Learning: Addressing Data Heterogeneity through Adaptive Normalization-Free Feature Recalibrationcited by 0
  4. 2025Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learningcited by 0
  5. 2025Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learningcited by 0
  6. 2025Technical note on Fisher Information for Robust Federated Cross-Validationcited by 0
  7. 2026Personalized Observation Normalization for Federated Reinforcement Learning in Simulation Environments with Heterogeneitycited by 0