# Experimenting with Normalization Layers in Federated Learning on non-IID scenarios

arXiv: https://arxiv.org/abs/2303.10630 · HTML: https://mlsanity.com/paper/2303.10630/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 19 Mar 2023.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models](https://mlsanity.com/paper/1702.03275/) (2017)
- [Overcoming Forgetting in Federated Learning on Non-IID Data](https://mlsanity.com/paper/1910.07796/) (2019)
- [Siloed Federated Learning for Multi-Centric Histopathology Datasets](https://mlsanity.com/paper/2008.07424/) (2020)
- [Efficient Decentralized Deep Learning by Dynamic Model Averaging](https://mlsanity.com/paper/1807.03210/) (2018)
- [Benchopt: Reproducible, efficient and collaborative optimization benchmarks](https://mlsanity.com/paper/2206.13424/) (2022)
- [Transfer Learning via Test-Time Neural Networks Aggregation](https://mlsanity.com/paper/2206.13399/) (2022)

## Built on by (most cited first, 8 of 8)

- [Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directions](https://mlsanity.com/paper/2411.12377/) (2024)
- [FedWon: Triumphing Multi-domain Federated Learning Without Normalization](https://mlsanity.com/paper/2306.05879/) (2023)
- [Understanding the Role of Layer Normalization in Label-Skewed Federated Learning](https://mlsanity.com/paper/2308.09565/) (2023)
- [The Effects of Data Imbalance Under a Federated Learning Approach for Credit Risk Forecasting](https://mlsanity.com/paper/2401.07234/) (2024)
- [Exploring the Efficacy of Group-Normalization in Deep Learning Models for Alzheimer's Disease Classification](https://mlsanity.com/paper/2404.00946/) (2024)
- [Feasibility Analysis of Federated Neural Networks for Explainable Detection of Atrial Fibrillation](https://mlsanity.com/paper/2410.19781/) (2024)
- [Federated Learning for MRI-based BrainAGE: a multicenter study on post-stroke functional outcome prediction](https://mlsanity.com/paper/2506.15626/) (2025)
- [Multi-Worker Selection based Distributed Swarm Learning for Edge IoT with Non-i.i.d. Data](https://mlsanity.com/paper/2509.18367/) (2025)

## Abstract (arXiv)

Training Deep Learning (DL) models require large, high-quality datasets, often assembled with data from different institutions. Federated Learning (FL) has been emerging as a method for privacy-preserving pooling of datasets employing collaborative training from different institutions by iteratively globally aggregating locally trained models. One critical performance challenge of FL is operating on datasets not independently and identically distributed (non-IID) among the federation participants. Even though this fragility cannot be eliminated, it can be debunked by a suitable optimization of two hyper-parameters: layer normalization methods and collaboration frequency selection. In this work, we benchmark five different normalization layers for training Neural Networks (NNs), two families of non-IID data skew, and two datasets. Results show that Batch Normalization, widely employed for centralized DL, is not the best choice for FL, whereas Group and Layer Normalization consistently outperform Batch Normalization. Similarly, frequent model aggregation decreases convergence speed and mode quality.