mlsanityData to 13 Oct 2026

FedTSA: A Cluster-based Two-Stage Aggregation Method for Model-heterogeneous Federated Learning

arXiv 2407.05098 ↗First posted 6 Jul 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Despite extensive research into data heterogeneity in federated learning (FL), system heterogeneity remains a significant yet often overlooked challenge. Traditional FL approaches typically assume homogeneous hardware resources across FL clients, implying that clients can train a global model within a comparable time frame. However, in practical FL systems, clients often have heterogeneous resources, which impacts their training capacity. This discrepancy underscores the importance of exploring model-heterogeneous FL, a paradigm allowing clients to train different models based on their resource capabilities. To address this challenge, we introduce FedTSA, a cluster-based two-stage aggregation method tailored for system heterogeneity in FL. FedTSA begins by clustering clients based on their capabilities, then performs a two-stage aggregation: conventional weight averaging for homogeneous models in Stage 1, and deep mutual learning with a diffusion model for aggregating heterogeneous models in Stage 2. Extensive experiments demonstrate that FedTSA not only outperforms the baselines but also explores various factors influencing model performance, validating FedTSA as a promising approach for model-heterogeneous FL.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 6 Jul 2024.

Builds on

43 of 58 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,656
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,213
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,653
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,294
  10. 2017Prototypical Networks for Few-shot Learningcited by 3,011
  11. 2018Federated Optimization in Heterogeneous Networkscited by 2,981
  12. 2022Diffusion Models in Vision: A Surveycited by 1,012
31 more
  1. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 903
  2. 2019Towards Federated Learning at Scale: System Designcited by 830
  3. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  4. 2017Deep Mutual Learningcited by 624
  5. 2020What is the State of Neural Network Pruning?cited by 582
  6. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  7. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  8. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  9. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  10. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 376
  11. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
  12. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  13. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  14. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  15. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  16. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  17. 2021Towards Understanding Knowledge Distillationcited by 158
  18. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  19. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  20. 2019SAFA: a Semi-Asynchronous Protocol for Fast Federated Learning with Low Overheadcited by 82
  21. 2021Convolutional Neural Network Pruning with Structural Redundancy Reductioncited by 73
  22. 2020LDP-FL: Practical Private Aggregation in Federated Learning with Local Differential Privacycited by 64
  23. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
  24. 2021Cluster-driven Graph Federated Learning over Multiple Domainscited by 44
  25. 2023Exploring One-shot Semi-supervised Federated Learning with A Pre-trained Diffusion Modelcited by 33
  26. 2022DropNet: Reducing Neural Network Complexity via Iterative Pruningcited by 29
  27. 2023TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Trainingcited by 27
  28. 2024Multi-level Personalized Federated Learning on Heterogeneous and Long-Tailed Datacited by 14
  29. 2023Learning Cautiously in Federated Learning with Noisy and Heterogeneous Clientscited by 12
  30. 2023Federated Learning with Diffusion Models for Privacy-Sensitive Vision Taskscited by 10
  31. 2023A Survey on Model-heterogeneous Federated Learning: Problems, Methods, and Prospectscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Communication-Efficient Federated Learning With Data and Client Heterogeneity
  2. 2025FedPref: Federated Learning Across Heterogeneous Multi-objective Preferences
  3. 2025FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectors
  4. 2024FedShift: Robust Federated Learning Aggregation Scheme in Resource Constrained Environment via Weight Shifting
  5. 2023One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
  6. 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation
  7. 2026Tram-FL: Reducing Communication and Computation Costs through Sequential Model Circulation in Decentralized Federated Learning
  8. 2025OvA-LP: A Simple and Efficient Framework for Federated Learning on Non-IID Data