mlsanityData to 7 Oct 2026

DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics

arXiv 2211.10878 ↗First posted 20 Nov 2022cs.LG cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

The Federated Learning (FL) paradigm is known to face challenges under heterogeneous client data. Local training on non-iid distributed data results in deflected local optimum, which causes the client models drift further away from each other and degrades the aggregated global model's performance. A natural solution is to gather all client data onto the server, such that the server has a global view of the entire data distribution. Unfortunately, this reduces to regular training, which compromises clients' privacy and conflicts with the purpose of FL. In this paper, we put forth an idea to collect and leverage global knowledge on the server without hindering data privacy. We unearth such knowledge from the dynamics of the global model's trajectory. Specifically, we first reserve a short trajectory of global model snapshots on the server. Then, we synthesize a small pseudo dataset such that the model trained on it mimics the dynamics of the reserved global model trajectory. Afterward, the synthesized data is used to help aggregate the deflected clients into the global model. We name our method Dynafed, which enjoys the following advantages: 1) we do not rely on any external on-server dataset, which requires no additional cost for data collection; 2) the pseudo data can be synthesized in early communication rounds, which enables Dynafed to take effect early for boosting the convergence and stabilizing training; 3) the pseudo data only needs to be synthesized once and can be directly utilized on the server to help aggregation in subsequent rounds. Experiments across extensive benchmarks are conducted to showcase the effectiveness of Dynafed. We also provide insights and understanding of the underlying mechanism of our method.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 44 of its 60 references that are in the map. First posted on arXiv 20 Nov 2022.

Builds on

44 of 60 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  9. 2018Federated Learning with Non-IID Datacited by 1,099
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  11. 2019Towards Federated Learning at Scale: System Designcited by 826
  12. 2021Model-Contrastive Federated Learningcited by 762
32 more
  1. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
  2. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  3. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  4. 2020The Future of Digital Health with Federated Learningcited by 625
  5. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  6. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  7. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  8. 2020Dataset Condensation with Gradient Matchingcited by 510
  9. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  10. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  11. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  12. 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 222
  13. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  14. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  15. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  16. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  17. 2020Survey of Personalization Techniques for Federated Learningcited by 144
  18. 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 122
  19. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 119
  20. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  21. 2021Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Modelscited by 106
  22. 2021Gradient Inversion with Generative Image Priorcited by 101
  23. 2019Robust Federated Learning in a Heterogeneous Environmentcited by 88
  24. 2021Heterogeneity for the Win: One-Shot Federated Clusteringcited by 87
  25. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  26. 2020XOR Mixup: Privacy-Preserving Data Augmentation for One-Shot Federated Learningcited by 58
  27. 2022GradViT: Gradient Inversion of Vision Transformerscited by 52
  28. 2020Inverse Distance Aggregation for Federated Learning with Non-IID Datacited by 44
  29. 2020Mix2FLD: Downlink Federated Learning After Uplink Federated Distillation With Two-Way Mixupcited by 29
  30. 2021FedFace: Collaborative Learning of Face Recognition Modelcited by 26
  31. 2017Analysis of Biased Stochastic Gradient Descent Using Sequential Semidefinite Programscited by 10
  32. 2022Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 8; 2025: 5; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  2. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  3. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  4. 2024Dataset Distillation by Automatic Training Trajectoriescited by 26
  5. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
  6. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 24
  7. 2024One-shot Federated Learning via Synthetic Distiller-Distillate Communicationcited by 12
  8. 2023FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Thingscited by 8
  9. 2025Federated Learning for Diffusion Modelscited by 6
  10. 2025Toward Malicious Clients Detection in Federated Learningcited by 4
  11. 2023Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectivescited by 3
  12. 2025TGDD: Trajectory Guided Dataset Distillation with Balanced Distributioncited by 3
10 more
  1. 2024Dataset Distillation-based Hybrid Federated Learning on Non-IID Datacited by 2
  2. 2024Tackling Data Heterogeneity in Federated Time Series Forecastingcited by 2
  3. 2023AST: Effective Dataset Distillation through Alignment with Smooth and High-Quality Expert Trajectoriescited by 1
  4. 2024Federated Cross-Training Learners for Robust Generalization under Data Heterogeneitycited by 1
  5. 2024One-Shot Collaborative Data Distillationcited by 1
  6. 2026Gradient Compression May Hurt Generalization: A Remedy by Synthetic Data Guided Sharpness Aware Minimizationcited by 1
  7. 2024Recovering Global Data Distribution Locally in Federated Learningcited by 0
  8. 2024DistDD: Distributed Data Distillation Aggregation through Gradient Matchingcited by 0
  9. 2025WarmFed: Federated Learning with Warm-Start for Globalization and Personalization Via Personalized Diffusion Modelscited by 0
  10. 2026Defending Against Backdoor Attacks via Alignment Checking in Model-Contrastive Federated Learningcited by 0