mlsanityData to 7 Oct 2026

Spectral Co-Distillation for Personalized Federated Learning

arXiv 2401.17124 ↗First posted 29 Jan 2024cs.LG cs.NIcited by 8

Not on the map. Why?

Abstract (arXiv)

Personalized federated learning (PFL) has been widely investigated to address the challenge of data heterogeneity, especially when a single generic model is inadequate in satisfying the diverse performance requirements of local clients simultaneously. Existing PFL methods are inherently based on the idea that the relations between the generic global and personalized local models are captured by the similarity of model weights. Such a similarity is primarily based on either partitioning the model architecture into generic versus personalized components, or modeling client relationships via model weights. To better capture similar (yet distinct) generic versus personalized model representations, we propose textitspectral distillation, a novel distillation method based on model spectrum information. Building upon spectral distillation, we also introduce a co-distillation framework that establishes a two-way bridge between generic and personalized model training. Moreover, to utilize the local idle time in conventional PFL, we propose a wait-free local training protocol. Through extensive experiments on multiple datasets over diverse heterogeneous data settings, we demonstrate the outperformance and efficacy of our proposed spectral co-distillation method, as well as our wait-free training protocol.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 41 of its 57 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

41 of 57 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2018On the Spectral Bias of Neural Networkscited by 1,321
  6. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  8. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  9. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  10. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  11. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  12. 2019Federated Learning with Personalization Layerscited by 533
29 more
  1. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  2. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  3. 2021Towards Personalized Federated Learningcited by 462
  4. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  5. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  6. 2020Adaptive Personalized Federated Learningcited by 353
  7. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  8. 2021A Field Guide to Federated Optimizationcited by 312
  9. 2019Asynchronous Federated Optimizationcited by 264
  10. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  11. 2021Personalized Federated Learning using Hypernetworkscited by 210
  12. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  13. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  14. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  15. 2022Federated Learning with Partial Model Personalizationcited by 114
  16. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  17. 2021Personalized Federated Learning with Gaussian Processescited by 76
  18. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  19. 2022FedCorr: Multi-Stage Federated Learning for Label Noise Correctioncited by 61
  20. 2020FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measurecited by 45
  21. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  22. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 36
  23. 2022CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learningcited by 34
  24. 2022Switchable Online Knowledge Distillationcited by 29
  25. 2022SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communicationcited by 21
  26. 2018Knowledge Distillation with Feature Maps for Image Classificationcited by 17
  27. 2022PGFed: Personalize Each Client's Global Objective for Federated Learningcited by 13
  28. 2021Dynamic Attention-based Communication-Efficient Federated Learningcited by 9
  29. 2022Towards Federated Long-Tailed Learningcited by 7

Built on by

Cited by year

Citing papers per year: 2024: 6; 2025: 2.

8 papers.

  1. 2024Advances in Robust Federated Learning: A Survey with Heterogeneity Considerationscited by 12
  2. 2024FedSheafHN: Personalized Federated Learning on Graph-structured Datacited by 2
  3. 2024FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learningcited by 2
  4. 2024FedCache 2.0: Federated Edge Learning with Knowledge Caching and Dataset Distillationcited by 1
  5. 2024Decentralized Personalized Federated Learningcited by 1
  6. 2024FedDTPT: Federated Discrete and Transferable Prompt Tuning for Black-Box Large Language Modelscited by 1
  7. 2025CLoVE: Personalized Federated Learning through Clustering of Loss Vector Embeddingscited by 1
  8. 2025Asynchronous Federated Continual Segmentation with Evolving Clients and Label Spacescited by 0