Stabilizing and Improving Federated Learning with Non-IID Data and Client Dropout
Not on the map. Why?
Abstract (arXiv)
The label distribution skew induced data heterogeniety has been shown to be a significant obstacle that limits the model performance in federated learning, which is particularly developed for collaborative model training over decentralized data sources while preserving user privacy. This challenge could be more serious when the participating clients are in unstable circumstances and dropout frequently. Previous work and our empirical observations demonstrate that the classifier head for classification task is more sensitive to label skew and the unstable performance of FedAvg mainly lies in the imbalanced training samples across different classes. The biased classifier head will also impact the learning of feature representations. Therefore, maintaining a balanced classifier head is of significant importance for building a better global model. To this end, we propose a simple yet effective framework by introducing a prior-calibrated softmax function for computing the cross-entropy loss and a prototype-based feature augmentation scheme to re-balance the local training, which are lightweight for edge devices and can facilitate the global model aggregation. The improved model performance over existing baselines in the presence of non-IID data and client dropout is demonstrated by conducting extensive experiments on benchmark classification tasks.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 41 of its 62 references that are in the map. First posted on arXiv 11 Mar 2023.
Builds on
41 of 62 references.
References · 41
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
- 2018Federated Learning with Non-IID Datacited by 1,099
- 2021Model-Contrastive Federated Learningcited by 762
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
- 2020Personalized Federated Learning with Moreau Envelopescited by 612
- 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
- 2021Federated Learning Based on Dynamic Regularizationcited by 555
- 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
29 more
- 2020An Efficient Framework for Clustered Federated Learningcited by 505
- 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
- 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
- 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 397
- 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
- 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
- 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
- 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
- 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
- 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
- 2015Inverting Visual Representations with Convolutional Networkscited by 168
- 2020Survey of Personalization Techniques for Federated Learningcited by 144
- 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
- 2020Federated Visual Classification with Real-World Data Distributioncited by 126
- 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
- 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
- 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
- 2019SAFA: a Semi-Asynchronous Protocol for Fast Federated Learning with Low Overheadcited by 82
- 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
- 2021FedCM: Federated Learning with Client-level Momentumcited by 77
- 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
- 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
- 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
- 2020Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneitycited by 56
- 2020Posterior Re-calibration for Imbalanced Datasetscited by 53
- 2021GRNN: Generative Regression Neural Network -- A Data Leakage Attack for Federated Learningcited by 35
- 2020FLAME: Differentially Private Federated Learning in the Shuffle Modelcited by 27
- 2020FedADC: Accelerated Federated Learning with Drift Controlcited by 26
- 2021FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systemscited by 16
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.