mlsanityData to 14 Oct 2026

Distilling A Universal Expert from Clustered Federated Learning

arXiv 2506.20285 ↗First posted 25 Jun 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Clustered Federated Learning (CFL) addresses the challenges posed by non-IID data by training multiple group- or cluster-specific expert models. However, existing methods often overlook the shared information across clusters, which represents the generalizable knowledge valuable to all participants in the Federated Learning (FL) system. To overcome this limitation, this paper introduces a novel FL framework that distills a universal expert model from the knowledge of multiple clusters. This universal expert captures globally shared information across all clients and is subsequently distributed to each client as the initialization for the next round of model training. The proposed FL framework operates in three iterative steps: (1) local model training at each client, (2) cluster-specific model aggregation, and (3) universal expert distillation. This three-step learning paradigm ensures the preservation of fine-grained non-IID characteristics while effectively incorporating shared knowledge across clusters. Compared to traditional gradient-based aggregation methods, the distillation-based model aggregation introduces greater flexibility in handling model heterogeneity and reduces conflicts among cluster-specific experts. Extensive experimental results demonstrate the superior performance of the proposed method across various scenarios, highlighting its potential to advance the state of CFL by balancing personalized and shared knowledge more effectively.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

26 of 41 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  5. 2021Model-Contrastive Federated Learningcited by 766
  6. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  7. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  8. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  9. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  10. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  11. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  12. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 252
14 more
  1. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  2. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 138
  3. 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 113
  4. 2020Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learningcited by 78
  5. 2021Flexible Clustered Federated Learning for Client-Level Data Distribution Shiftcited by 47
  6. 2021FedSoft: Soft Clustered Federated Learning with Proximal Local Updatingcited by 46
  7. 2022Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspacescited by 36
  8. 2020Data-Free Knowledge Amalgamation via Group-Stack Dual-GANcited by 32
  9. 2022On the Convergence of Clustered Federated Learningcited by 31
  10. 2023Dual Personalization on Federated Recommendationcited by 19
  11. 2023FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clusteringcited by 15
  12. 2022SwiftAgg: Communication-Efficient and Dropout-Resistant Secure Aggregation for Federated Learning with Worst-Case Security Guaranteescited by 14
  13. 2024Bridging Model Heterogeneity in Federated Learning via Uncertainty-based Asymmetrical Reciprocity Learningcited by 11
  14. 2024Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Single-Round Clustered Federated Learning via Data Collaboration Analysis for Non-IID Datacited by 1