Unlocking the Potential of Federated Learning for Deeper Models
Not on the map. Why?
Abstract (arXiv)
Federated learning (FL) is a new paradigm for distributed machine learning that allows a global model to be trained across multiple clients without compromising their privacy. Although FL has demonstrated remarkable success in various scenarios, recent studies mainly utilize shallow and small neural networks. In our research, we discover a significant performance decline when applying the existing FL framework to deeper neural networks, even when client data are independently and identically distributed (i.i.d.). Our further investigation shows that the decline is due to the continuous accumulation of dissimilarities among client models during the layer-by-layer back-propagation process, which we refer to as "divergence accumulation." As deeper models involve a longer chain of divergence accumulation, they tend to manifest greater divergence, subsequently leading to performance decline. Both theoretical derivations and empirical evidence are proposed to support the existence of divergence accumulation and its amplified effects in deeper models. To address this issue, we propose several technical guidelines based on reducing divergence, such as using wider models and reducing the receptive field. These approaches can greatly improve the accuracy of FL on deeper models. For example, the application of these guidelines can boost the ResNet101 model's performance by as much as 43% on the Tiny-ImageNet dataset.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 5 Jun 2023.
Builds on
28 of 37 references.
References · 28
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2016Layer Normalizationcited by 5,579
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
- 2018LEAF: A Benchmark for Federated Settingscited by 780
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
16 more
- 2019Federated Learning with Personalization Layerscited by 533
- 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
- 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
- 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 178
- 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
- 2020Attribute Prototype Network for Zero-Shot Learningcited by 160
- 2018Robust Classification with Convolutional Prototype Learningcited by 122
- 2017Medical Image Analysis using Convolutional Neural Networks: A Reviewcited by 99
- 2021SparseFed: Mitigating Model Poisoning Attacks in Federated Learning with Sparsificationcited by 46
- 2020From Federated Learning to Federated Neural Architecture Search: A Surveycited by 42
- 2020Dynamic Fusion based Federated Learning for COVID-19 Detectioncited by 36
- 2021FedCV: A Federated Learning Framework for Diverse Computer Vision Taskscited by 32
- 2022Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferringcited by 28
- 2020Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimizationcited by 25
- 2020Real-time Federated Evolutionary Neural Architecture Searchcited by 20
- 2020Federated Neural Architecture Searchcited by 10
Built on by
Citing papers per year: 2024: 1.
1 paper.