# Distilling A Universal Expert from Clustered Federated Learning

arXiv: https://arxiv.org/abs/2506.20285 · HTML: https://mlsanity.com/paper/2506.20285/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 25 Jun 2025.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Federated Learning for Internet of Things: A Comprehensive Survey](https://mlsanity.com/paper/2104.07914/) (2021)
- [Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning](https://mlsanity.com/paper/2203.09249/) (2022)
- [Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges](https://mlsanity.com/paper/2211.08413/) (2022)
- [Multi-Center Federated Learning: Clients Clustering for Better Personalization](https://mlsanity.com/paper/2108.08647/) (2021)
- [Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning](https://mlsanity.com/paper/2002.04156/) (2020)
- [Flexible Clustered Federated Learning for Client-Level Data Distribution Shift](https://mlsanity.com/paper/2108.09749/) (2021)
- [FedSoft: Soft Clustered Federated Learning with Proximal Local Updating](https://mlsanity.com/paper/2112.06053/) (2021)
- [Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspaces](https://mlsanity.com/paper/2209.10526/) (2022)
- [Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN](https://mlsanity.com/paper/2003.09088/) (2020)
- [On the Convergence of Clustered Federated Learning](https://mlsanity.com/paper/2202.06187/) (2022)
- [Dual Personalization on Federated Recommendation](https://mlsanity.com/paper/2301.08143/) (2023)
- [FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering](https://mlsanity.com/paper/2301.12379/) (2023)
- [SwiftAgg: Communication-Efficient and Dropout-Resistant Secure Aggregation for Federated Learning with Worst-Case Security Guarantees](https://mlsanity.com/paper/2202.04169/) (2022)
- [Bridging Model Heterogeneity in Federated Learning via Uncertainty-based Asymmetrical Reciprocity Learning](https://mlsanity.com/paper/2407.03247/) (2024)
- [Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation](https://mlsanity.com/paper/2402.12406/) (2024)

## Built on by (most cited first, 1 of 1)

- [Single-Round Clustered Federated Learning via Data Collaboration Analysis for Non-IID Data](https://mlsanity.com/paper/2601.09304/) (2026)

## Abstract (arXiv)

Clustered Federated Learning (CFL) addresses the challenges posed by non-IID data by training multiple group- or cluster-specific expert models. However, existing methods often overlook the shared information across clusters, which represents the generalizable knowledge valuable to all participants in the Federated Learning (FL) system. To overcome this limitation, this paper introduces a novel FL framework that distills a universal expert model from the knowledge of multiple clusters. This universal expert captures globally shared information across all clients and is subsequently distributed to each client as the initialization for the next round of model training. The proposed FL framework operates in three iterative steps: (1) local model training at each client, (2) cluster-specific model aggregation, and (3) universal expert distillation. This three-step learning paradigm ensures the preservation of fine-grained non-IID characteristics while effectively incorporating shared knowledge across clusters. Compared to traditional gradient-based aggregation methods, the distillation-based model aggregation introduces greater flexibility in handling model heterogeneity and reduces conflicts among cluster-specific experts. Extensive experimental results demonstrate the superior performance of the proposed method across various scenarios, highlighting its potential to advance the state of CFL by balancing personalized and shared knowledge more effectively.