mlsanityData to 7 Oct 2026

Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchors

arXiv 2405.11525 ↗First posted 19 May 2024cs.LG cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Conventional Federated Learning (FL) involves collaborative training of a global model while maintaining user data privacy. One of its branches, decentralized FL, is a serverless network that allows clients to own and optimize different local models separately, which results in saving management and communication resources. Despite the promising advancements in decentralized FL, it may reduce model generalizability due to lacking a global model. In this scenario, managing data and model heterogeneity among clients becomes a crucial problem, which poses a unique challenge that must be overcome: How can every client's local model learn generalizable representation in a decentralized manner? To address this challenge, we propose a novel Decentralized FL technique by introducing Synthetic Anchors, dubbed as DeSA. Based on the theory of domain adaptation and Knowledge Distillation (KD), we theoretically and empirically show that synthesizing global anchors based on raw data distribution facilitates mutual knowledge transfer. We further design two effective regularization terms for local training: 1) REG loss that regularizes the distribution of the client's latent embedding with the anchors and 2) KD loss that enables clients to learn from others. Through extensive experiments on diverse client data distributions, we showcase the effectiveness of DeSA in enhancing both inter- and intra-domain accuracy of each client.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 60 of its 84 references that are in the map. First posted on arXiv 19 May 2024.

Builds on

60 of 84 references.

References · 60

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2016Deep Learning with Differential Privacycited by 2,682
  8. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  9. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  10. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  11. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  12. 2021Membership Inference Attacks From First Principlescited by 839
38 more
  1. 2021Model-Contrastive Federated Learningcited by 762
  2. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  3. 2017Deep Mutual Learningcited by 621
  4. 2020Dataset Condensation with Gradient Matchingcited by 510
  5. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  6. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  7. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  8. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  9. 2021Dataset Condensation with Distribution Matchingcited by 362
  10. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  11. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  12. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
  13. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  14. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  15. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 189
  16. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  17. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 180
  18. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  19. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  20. 2018Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentationcited by 148
  21. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 137
  22. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 119
  23. 2019BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learningcited by 115
  24. 2019Generalizing to unseen domains via distribution matchingcited by 109
  25. 2021Fair Mixup: Fairness via Interpolationcited by 109
  26. 2022Dataset Condensation with Contrastive Signalscited by 105
  27. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  28. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
  29. 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
  30. 2022Synthesizing Informative Training Samples with GANcited by 77
  31. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 65
  32. 2020Model-sharing Games: Analyzing Federated Learning Under Voluntary Participationcited by 60
  33. 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
  34. 2020KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillationcited by 50
  35. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  36. 2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 49
  37. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
  38. 2020Decentralized Federated Learning via Mutual Knowledge Transfercited by 42

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 7.

21 papers.

Sort:

  1. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 24
  2. 2025TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendationcited by 13
  3. 2024FedVCK: Non-IID Robust and Communication-Efficient Federated Learning via Valuable Condensed Knowledge for Medical Image Analysiscited by 6
  4. 2025FedWSIDD: Federated Whole Slide Image Classification via Dataset Distillationcited by 3
  5. 2024Dataset Distillation-based Hybrid Federated Learning on Non-IID Datacited by 2
  6. 2025Parameterizing Dataset Distillation via Gaussian Splattingcited by 2
  7. 2026Toward Enhancing Representation Learning in Federated Multi-Task Settingscited by 2
  8. 2025Robust Dataset Distillation by Matching Adversarial Trajectoriescited by 1
  9. 2025FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classificationcited by 1
  10. 2025Bridging Generalization Gap of Heterogeneous Federated Clients Using Generative Modelscited by 1
  11. 2025On the Fast Adaptation of Delayed Clients in Decentralized Federated Learning: A Centroid-Aligned Distillation Approachcited by 1
  12. 2026DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recoverycited by 1
9 more
  1. 2025ICAFS: Inter-Client-Aware Feature Selection for Vertical Federated Learningcited by 0
  2. 2025Game-theoretic distributed learning of generative models for heterogeneous data collectionscited by 0
  3. 2025A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learningcited by 0
  4. 2025DiRe: Diversity-promoting Regularization for Dataset Condensationcited by 0
  5. 2026DecHW: Heterogeneous Decentralized Federated Learning Exploiting Second-Order Informationcited by 0
  6. 2026Federated Distillation for Whole Slide Image via Gaussian-Mixture Feature Alignment and Curriculum Integrationcited by 0
  7. 2026Learned Neighbor Trust for Collaborative Deployment in Model-Agnostic Decentralized Learningcited by 0
  8. 2026TallyTrain: Communication-Efficient Federated Distillationcited by 0
  9. 2026Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Networkcited by 0