# FedD2S: Personalized Data-Free Federated Knowledge Distillation

arXiv: https://arxiv.org/abs/2402.10846 · HTML: https://mlsanity.com/paper/2402.10846/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 26 of 26)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017)
- [Data-Free Adversarial Distillation](https://mlsanity.com/paper/1912.11006/) (2019)
- [Client Selection in Federated Learning: Principles, Challenges, and Opportunities](https://mlsanity.com/paper/2211.01549/) (2022)
- [Flexible Clustered Federated Learning for Client-Level Data Distribution Shift](https://mlsanity.com/paper/2108.09749/) (2021)
- [FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models](https://mlsanity.com/paper/2109.03775/) (2021)
- [MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing](https://mlsanity.com/paper/2011.09899/) (2020)
- [The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation](https://mlsanity.com/paper/2301.08968/) (2023)
- [HiFlash: Communication-Efficient Hierarchical Federated Learning with Adaptive Staleness Control and Heterogeneity-aware Client-Edge Association](https://mlsanity.com/paper/2301.06447/) (2023)
- [FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks](https://mlsanity.com/paper/2201.03169/) (2022)
- [Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression](https://mlsanity.com/paper/2012.03096/) (2020)
- [Leveraging Angular Distributions for Improved Knowledge Distillation](https://mlsanity.com/paper/2302.14130/) (2023)

## Built on by (most cited first, 3 of 3)

- [Leveraging Foundation Models for Efficient Federated Learning in Resource-restricted Edge Networks](https://mlsanity.com/paper/2409.09273/) (2024)
- [KnFu: Effective Knowledge Fusion](https://mlsanity.com/paper/2403.11892/) (2024)
- [Whom to Trust? Adaptive Collaboration in Personalized Federated Learning](https://mlsanity.com/paper/2507.00259/) (2025)

## Abstract (arXiv)

This paper addresses the challenge of mitigating data heterogeneity among clients within a Federated Learning (FL) framework. The model-drift issue, arising from the noniid nature of client data, often results in suboptimal personalization of a global model compared to locally trained models for each client. To tackle this challenge, we propose a novel approach named FedD2S for Personalized Federated Learning (pFL), leveraging knowledge distillation. FedD2S incorporates a deep-to-shallow layer-dropping mechanism in the data-free knowledge distillation process to enhance local model personalization. Through extensive simulations on diverse image datasets-FEMNIST, CIFAR10, CINIC0, and CIFAR100-we compare FedD2S with state-of-the-art FL baselines. The proposed approach demonstrates superior performance, characterized by accelerated convergence and improved fairness among clients. The introduced layer-dropping technique effectively captures personalized knowledge, resulting in enhanced performance compared to alternative FL models. Moreover, we investigate the impact of key hyperparameters, such as the participation ratio and layer-dropping rate, providing valuable insights into the optimal configuration for FedD2S. The findings demonstrate the efficacy of adaptive layer-dropping in the knowledge distillation process to achieve enhanced personalization and performance across diverse datasets and tasks.