# CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learning

arXiv: https://arxiv.org/abs/2204.03880 · HTML: https://mlsanity.com/paper/2204.03880/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 8 Apr 2022.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space](https://mlsanity.com/paper/2103.06030/) (2021)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [Federated Learning of a Mixture of Global and Local Models](https://mlsanity.com/paper/2002.05516/) (2020)
- [Survey of Personalization Techniques for Federated Learning](https://mlsanity.com/paper/2003.08673/) (2020)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Salvaging Federated Learning by Local Adaptation](https://mlsanity.com/paper/2002.04758/) (2020)

## Built on by (most cited first, 30 of 34)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation](https://mlsanity.com/paper/2308.15367/) (2023)
- [Federated Domain Generalization: A Survey](https://mlsanity.com/paper/2306.01334/) (2023)
- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations](https://mlsanity.com/paper/2405.09839/) (2024)
- [FedWon: Triumphing Multi-domain Federated Learning Without Normalization](https://mlsanity.com/paper/2306.05879/) (2023)
- [FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data](https://mlsanity.com/paper/2401.08977/) (2024)
- [FedUV: Uniformity and Variance for Heterogeneous Federated Learning](https://mlsanity.com/paper/2402.18372/) (2024)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [Spectral Co-Distillation for Personalized Federated Learning](https://mlsanity.com/paper/2401.17124/) (2024)
- [A Systematic Review of Generalization Research in Medical Image Classification](https://mlsanity.com/paper/2403.12167/) (2024)
- [Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2312.13923/) (2023)
- [Influence-Oriented Personalized Federated Learning](https://mlsanity.com/paper/2410.03315/) (2024)
- [Personalized Federated Learning with Communication Compression](https://mlsanity.com/paper/2209.05148/) (2022)
- [Federated Learning of Shareable Bases for Personalization-Friendly Image Classification](https://mlsanity.com/paper/2304.07882/) (2023)
- [Federated brain tumor segmentation: an extensive benchmark](https://mlsanity.com/paper/2410.17265/) (2024)
- [A Client-server Deep Federated Learning for Cross-domain Surgical Image Segmentation](https://mlsanity.com/paper/2306.08720/) (2023)
- [UNIDEAL: Curriculum Knowledge Distillation Federated Learning](https://mlsanity.com/paper/2309.08961/) (2023)
- [Spatio-Temporal Representation Decoupling and Enhancement for Federated Instrument Segmentation in Surgical Videos](https://mlsanity.com/paper/2506.23759/) (2025)
- [Personalizing Federated Instrument Segmentation with Visual Trait Priors in Robotic Surgery](https://mlsanity.com/paper/2408.03208/) (2024)
- [Federated Client-tailored Adapter for Medical Image Segmentation](https://mlsanity.com/paper/2504.18020/) (2025)
- [Robust Knowledge Adaptation for Federated Unsupervised Person ReID](https://mlsanity.com/paper/2301.07320/) (2023)
- [DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representations](https://mlsanity.com/paper/2407.17754/) (2024)
- [Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generator](https://mlsanity.com/paper/2409.06955/) (2024)
- [Asymmetrical Reciprocity-based Federated Learning for Resolving Disparities in Medical Diagnosis](https://mlsanity.com/paper/2412.19654/) (2024)
- [Communication-Efficient and Personalized Federated Foundation Model Fine-Tuning via Tri-Matrix Adaptation](https://mlsanity.com/paper/2503.23869/) (2025)
- [FedPartWhole: Federated domain generalization via consistent part-whole hierarchies](https://mlsanity.com/paper/2407.14792/) (2024)
- [Personalized Federated Learning on Heterogeneous and Long-Tailed Data via Expert Collaborative Learning](https://mlsanity.com/paper/2408.02019/) (2024)
- [One-Time Model Adaptation to Heterogeneous Clients: An Intra-Client and Inter-Image Attention Design](https://mlsanity.com/paper/2211.06276/) (2022)
- [Factor-Assisted Federated Learning for Personalized Optimization with Heterogeneous Data](https://mlsanity.com/paper/2312.04281/) (2023)

## Abstract (arXiv)

Federated learning (FL) is a distributed learning paradigm that enables multiple clients to collaboratively learn a shared global model. Despite the recent progress, it remains challenging to deal with heterogeneous data clients, as the discrepant data distributions usually prevent the global model from delivering good generalization ability on each participating client. In this paper, we propose CD^2-pFed, a novel Cyclic Distillation-guided Channel Decoupling framework, to personalize the global model in FL, under various settings of data heterogeneity. Different from previous works which establish layer-wise personalization to overcome the non-IID data across different clients, we make the first attempt at channel-wise assignment for model personalization, referred to as channel decoupling. To further facilitate the collaboration between private and shared weights, we propose a novel cyclic distillation scheme to impose a consistent regularization between the local and global model representations during the federation. Guided by the cyclical distillation, our channel decoupling framework can deliver more accurate and generalized results for different kinds of heterogeneity, such as feature skew, label distribution skew, and concept shift. Comprehensive experiments on four benchmarks, including natural image and medical image analysis tasks, demonstrate the consistent effectiveness of our method on both local and external validations.