mlsanityData to 7 Oct 2026

One-shot Federated Learning via Synthetic Distiller-Distillate Communication

arXiv 2412.05186 ↗First posted 6 Dec 2024cs.LG cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

One-shot Federated learning (FL) is a powerful technology facilitating collaborative training of machine learning models in a single round of communication. While its superiority lies in communication efficiency and privacy preservation compared to iterative FL, one-shot FL often compromises model performance. Prior research has primarily focused on employing data-free knowledge distillation to optimize data generators and ensemble models for better aggregating local knowledge into the server model. However, these methods typically struggle with data heterogeneity, where inconsistent local data distributions can cause teachers to provide misleading knowledge. Additionally, they may encounter scalability issues with complex datasets due to inherent two-step information loss: first, during local training (from data to model), and second, when transferring knowledge to the server model (from model to inversed data). In this paper, we propose FedSD2C, a novel and practical one-shot FL framework designed to address these challenges. FedSD2C introduces a distiller to synthesize informative distillates directly from local data to reduce information loss and proposes sharing synthetic distillates instead of inconsistent local models to tackle data heterogeneity. Our empirical results demonstrate that FedSD2C consistently outperforms other one-shot FL methods with more complex and real datasets, achieving up to 2.6 the performance of the best baseline. Code: https://github.com/Carkham/FedSD2C

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 52 of its 66 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

52 of 66 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  8. 2018Federated Learning with Non-IID Datacited by 1,099
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  10. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  11. 2020Dataset Condensation with Gradient Matchingcited by 510
  12. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
38 more
  1. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  2. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  3. 2021Dataset Condensation with Distribution Matchingcited by 362
  4. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  5. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
  6. 2021A Fourier-based Framework for Domain Generalizationcited by 287
  7. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  8. 2019Model Fusion via Optimal Transportcited by 217
  9. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  10. 2019Data-Free Learning of Student Networkscited by 203
  11. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  12. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  13. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 136
  14. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  15. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  16. 2019One-Shot Federated Learningcited by 124
  17. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 119
  18. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  19. 2020Distilled One-Shot Federated Learningcited by 108
  20. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  21. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  22. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  23. 2022Deep Model Reassemblycited by 78
  24. 2020Practical One-Shot Federated Learning for Cross-Silo Settingcited by 76
  25. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 65
  26. 2022Factorizing Knowledge in Neural Networkscited by 59
  27. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  28. 2022Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imagingcited by 39
  29. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 35
  30. 2024Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boostingcited by 27
  31. 2022Meta Knowledge Condensation for Federated Learningcited by 26
  32. 2023Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teachercited by 25
  33. 2023Federated Generative Learning with Foundation Modelscited by 23
  34. 2022DYNAFED: Tackling Client Data Heterogeneity with Global Dynamicscited by 22
  35. 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
  36. 2019Man-in-the-Middle Attacks against Machine Learning Classifiers via Malicious Generative Modelscited by 19
  37. 2024An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learningcited by 17
  38. 2024Teddy: Efficient Large-Scale Dataset Distillation via Taylor-Approximated Matchingcited by 13

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025One-shot Federated Learning Methods: A Practical Guidecited by 6
  2. 2025Towards One-shot Federated Learning: Advances, Challenges, and Future Directionscited by 2
  3. 2025FedMHO: Heterogeneous One-Shot Federated Learning Towards Resource-Constrained Edge Devicescited by 1
  4. 2025On the Fast Adaptation of Delayed Clients in Decentralized Federated Learning: A Centroid-Aligned Distillation Approachcited by 1
  5. 2025TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Modelscited by 1
  6. 2026Distribution Alignment for One-Shot Federated Learning via Optimal Transportcited by 1
  7. 2025Single-Round Scalable Analytic Federated Learningcited by 0
  8. 2026Feature-Aware One-Shot Federated Learning via Hierarchical Token Sequencescited by 0
  9. 2026Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learningcited by 0
  10. 2026FedOPAL: One-Shot Federated Learning via Analytic Visual Prompt Tuningcited by 0
  11. 2026CRIP: Channel Level Representation Injection for Personalized One-Shot Federated Learningcited by 0
  12. 2026Structural Entropy-Driven Graph Diffusion Generation for One-Shot Federated Graph Learningcited by 0