mlsanityData to 7 Oct 2026

FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

arXiv 2102.07623 ↗First posted 15 Feb 2021cs.LGcited by 565

Not on the map. Why?

Abstract (arXiv)

The emerging paradigm of federated learning (FL) strives to enable collaborative training of deep models on the network edge without centrally aggregating raw data and hence improving data privacy. In most cases, the assumption of independent and identically distributed samples across local clients does not hold for federated learning setups. Under this setting, neural network training performance may vary significantly according to the data distribution and even hurt training convergence. Most of the previous work has focused on a difference in the distribution of labels or client shifts. Unlike those settings, we address an important problem of FL, e.g., different scanners/sensors in medical imaging, different scenery distribution in autonomous driving (highway vs. city), where local clients store examples with different distributions compared to other clients, which we denote as feature shift non-iid. In this work, we propose an effective method that uses local batch normalization to alleviate the feature shift before averaging models. The resulting scheme, called FedBN, outperforms both classical FedAvg, as well as the state-of-the-art for non-iid data (FedProx) on our extensive experiments. These empirical results are supported by a convergence analysis that shows in a simplified setting that FedBN has a faster convergence rate than FedAvg. Code is available at https://github.com/med-air/FedBN.

Key facts. Cited by 565 papers in the map (data through 7 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 15 Feb 2021.

Builds on

36 of 51 references.

References · 36

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  7. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  9. 2018Federated Learning with Non-IID Datacited by 1,099
  10. 2020Adaptive Federated Optimizationcited by 1,036
  11. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  12. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
24 more
  1. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  2. 2020The Future of Digital Health with Federated Learningcited by 625
  3. 2020Federated Learning with Matched Averagingcited by 564
  4. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  5. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  6. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  7. 2018How Does Batch Normalization Help Optimization?cited by 355
  8. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  9. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  10. 2016Revisiting Batch Normalization For Practical Domain Adaptationcited by 285
  11. 2019Domain-Specific Batch Normalization for Unsupervised Domain Adaptationcited by 197
  12. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  13. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
  14. 2020MS-Net: Multi-Site Network for Improving Prostate Segmentation with Heterogeneous MRI Datacited by 121
  15. 2018A generic framework for privacy preserving deep learningcited by 103
  16. 2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
  17. 2018On the importance of single directions for generalizationcited by 99
  18. 2020Siloed Federated Learning for Multi-Centric Histopathology Datasetscited by 61
  19. 2020Training (Overparametrized) Neural Networks in Near-Linear Timecited by 58
  20. 2018Towards Understanding Regularization in Batch Normalizationcited by 56
  21. 2018Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learningcited by 56
  22. 2018Efficient Decentralized Deep Learning by Dynamic Model Averagingcited by 30
  23. 2018Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimizationcited by 27
  24. 2020Optimization Theory for ReLU Neural Networks Trained with Normalization Layerscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 40; 2022: 95; 2023: 133; 2024: 124; 2025: 99; 2026: 74.

565 papers.

Sort:

By area:

  1. 2021Model-Contrastive Federated Learningcited by 762
  2. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  3. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  4. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 189
  5. 2021Federated Learning for Smart Healthcare: A Surveycited by 182
  6. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  7. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  8. 2022Federated Class-Incremental Learningcited by 126
  9. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
  10. 2021Gradient Inversion with Generative Image Priorcited by 101
  11. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  12. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
38 more
  1. 2022Federated Learning on Non-IID Graphs via Structural Knowledge Sharingcited by 88
  2. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  3. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  4. 2021HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Imagescited by 62
  5. 2021FedLab: A Flexible Federated Learning Frameworkcited by 54
  6. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  7. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  8. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 54
  9. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  10. 2023No One Left Behind: Real-World Federated Class-Incremental Learningcited by 53
  11. 2023Fair Federated Medical Image Segmentation via Client Contribution Estimationcited by 50
  12. 2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 49
  13. 2021SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Trainingcited by 45
  14. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 45
  15. 2022Do Gradient Inversion Attacks Make Federated Learning Unsafe?cited by 44
  16. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  17. 2022FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated Graph Learningcited by 43
  18. 2022FedDrive: Generalizing Federated Learning to Semantic Segmentation in Autonomous Drivingcited by 41
  19. 2022Federated Domain Generalization for Image Recognition via Cross-Client Style Transfercited by 40
  20. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  21. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  22. 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 39
  23. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
  24. 2022MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcarecited by 36
  25. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  26. 2023Federated Incremental Semantic Segmentationcited by 36
  27. 2021Specificity-Preserving Federated Learning for MR Image Reconstructioncited by 35
  28. 2022Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customizationcited by 34
  29. 2022CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learningcited by 34
  30. 2022Federated Learning of Generative Image Priors for MRI Reconstructioncited by 33
  31. 2023FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise Heterogeneitycited by 33
  32. 2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
  33. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  34. 2021FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimizationcited by 29
  35. 2022Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learningcited by 29
  36. 2023Why Batch Normalization Damage Federated Learning on Non-IID Data?cited by 29
  37. 2022Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferringcited by 28
  38. 2022Personalizing Federated Medical Image Segmentation via Local Calibrationcited by 28

50 most cited of 565 All 565 (JSON)