Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity
Not on the map. Why?
Abstract (arXiv)
In federated learning, data heterogeneity is a critical challenge. A straightforward solution is to shuffle the clients' data to homogenize the distribution. However, this may violate data access rights, and how and when shuffling can accelerate the convergence of a federated optimization algorithm is not theoretically well understood. In this paper, we establish a precise and quantifiable correspondence between data heterogeneity and parameters in the convergence rate when a fraction of data is shuffled across clients. We prove that shuffling can quadratically reduce the gradient dissimilarity with respect to the shuffling percentage, accelerating convergence. Inspired by the theory, we propose a practical approach that addresses the data access rights issue by shuffling locally generated synthetic data. The experimental results show that shuffling synthetic data improves the performance of multiple existing federated learning algorithms by a large margin.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 44 of its 61 references that are in the map. First posted on arXiv 23 Jun 2023.
Builds on
44 of 61 references.
References · 44
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2016Improved Techniques for Training GANscited by 2,938
- 2016Deep Learning with Differential Privacycited by 2,682
- 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
- 2018Federated Learning with Non-IID Datacited by 1,099
- 2020Adaptive Federated Optimizationcited by 1,036
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
- 2021Model-Contrastive Federated Learningcited by 762
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
32 more
- 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
- 2021Federated Learning Based on Dynamic Regularizationcited by 555
- 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
- 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
- 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
- 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
- 2021A Field Guide to Federated Optimizationcited by 312
- 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 287
- 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
- 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
- 2018Differentially Private Generative Adversarial Networkcited by 235
- 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
- 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
- 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
- 2022Differentially Private Diffusion Modelscited by 118
- 2020GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generatorscited by 117
- 2021Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?cited by 113
- 2019Distributed Learning with Compressed Gradient Differencescited by 110
- 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
- 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
- 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
- 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 81
- 2021On Large-Cohort Training for Federated Learningcited by 73
- 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
- 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 65
- 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
- 2020Federated Learning via Synthetic Datacited by 53
- 2021Negative Data Augmentationcited by 46
- 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 36
- 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
- 2022Federated Learning with GAN-based Data Synthesis for Non-IID Clientscited by 25
- 2022On the effectiveness of partial variance reduction in federated learning with heterogeneous datacited by 8
Built on by
Citing papers per year: 2024: 2.
2 papers.