FedPeWS: Personalized Warmup via Subnetworks for Enhanced Heterogeneous Federated Learning
Not on the map. Why?
Abstract (arXiv)
Statistical data heterogeneity is a significant barrier to convergence in federated learning (FL). While prior work has advanced heterogeneous FL through better optimization objectives, these methods fall short when there is extreme data heterogeneity among collaborating participants. We hypothesize that convergence under extreme data heterogeneity is primarily hindered due to the aggregation of conflicting updates from the participants in the initial collaboration rounds. To overcome this problem, we propose a warmup phase where each participant learns a personalized mask and updates only a subnetwork of the full model. This personalized warmup allows the participants to focus initially on learning specific subnetworks tailored to the heterogeneity of their data. After the warmup phase, the participants revert to standard federated optimization, where all parameters are communicated. We empirically demonstrate that the proposed personalized warmup via subnetworks (FedPeWS) approach improves accuracy and convergence speed over standard federated optimization methods.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 44 of its 56 references that are in the map. First posted on arXiv 3 Oct 2024.
Builds on
44 of 56 references.
References · 44
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
- 2021Model-Contrastive Federated Learningcited by 762
- 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
- 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
- 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
32 more
- 2020Federated Learning with Matched Averagingcited by 564
- 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
- 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
- 2020Neural Architecture Search without Trainingcited by 241
- 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
- 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
- 2020Federated Continual Learning with Weighted Inter-client Transfercited by 158
- 2018RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasetscited by 148
- 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
- 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
- 2019First Analysis of Local GD on Heterogeneous Datacited by 80
- 2020Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithmscited by 78
- 2020Practical One-Shot Federated Learning for Cross-Silo Settingcited by 76
- 2019MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Samplingcited by 67
- 2020LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasetscited by 60
- 2023Fair Federated Medical Image Segmentation via Client Contribution Estimationcited by 50
- 2020Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learningcited by 37
- 2021Personalized Federated Learning: A Unified Framework and Universal Optimization Techniquescited by 34
- 2021CaPC Learning: Confidential and Private Collaborative Learningcited by 33
- 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
- 2022Achieving Personalized Federated Learning with Sparse Local Modelscited by 30
- 2019Distributed Learning of Deep Neural Networks using Independent Subnet Trainingcited by 20
- 2023Federated Learning as Variational Inference: A Scalable Expectation Propagation Approachcited by 17
- 2020Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learningcited by 16
- 2024Redefining Contributions: Shapley-Driven Federated Learningcited by 16
- 2021Decentralized Personalized Federated Learning: Lower Bounds and Optimal Algorithm for All Personalization Modescited by 13
- 2021FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learningcited by 12
- 2021FRL: Federated Rank Learningcited by 8
- 2023EvoFed: Leveraging Evolutionary Strategies for Communication-Efficient Federated Learningcited by 6
- 2023Private Federated Learning with Autotuned Compressioncited by 5
- 2024Federated Learning Can Find Friends That Are Advantageouscited by 1
- 2024Local Methods with Adaptivity via Scalingcited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.