# Federated Model Distillation with Noise-Free Differential Privacy

arXiv: https://arxiv.org/abs/2009.05537 · HTML: https://mlsanity.com/paper/2009.05537/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2020.

## Built on by (most cited first, 30 of 40)

- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [FedKD: Communication Efficient Federated Learning via Knowledge Distillation](https://mlsanity.com/paper/2108.13323/) (2021)
- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [Distributed Learning in Wireless Networks: Recent Progress and Future Challenges](https://mlsanity.com/paper/2104.02151/) (2021)
- [Parameterized Knowledge Transfer for Personalized Federated Learning](https://mlsanity.com/paper/2111.02862/) (2021)
- [DENSE: Data-Free One-Shot Federated Learning](https://mlsanity.com/paper/2112.12371/) (2021)
- [FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning](https://mlsanity.com/paper/2102.02514/) (2021)
- [Decentralized Federated Learning through Proxy Model Sharing](https://mlsanity.com/paper/2111.11343/) (2021)
- [No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices](https://mlsanity.com/paper/2202.08036/) (2022)
- [Differentially Private Federated Learning with Local Regularization and Sparsification](https://mlsanity.com/paper/2203.03106/) (2022)
- [Source Inference Attacks in Federated Learning](https://mlsanity.com/paper/2109.05659/) (2021)
- [Make Landscape Flatter in Differentially Private Federated Learning](https://mlsanity.com/paper/2303.11242/) (2023)
- [FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models](https://mlsanity.com/paper/2109.03775/) (2021)
- [Towards Personalized Federated Learning via Heterogeneous Model Reassembly](https://mlsanity.com/paper/2308.08643/) (2023)
- [Decentralized Federated Learning: A Survey on Security and Privacy](https://mlsanity.com/paper/2401.17319/) (2024)
- [Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer](https://mlsanity.com/paper/2109.08119/) (2021)
- [A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency](https://mlsanity.com/paper/2307.10655/) (2023)
- [OpenFGL: A Comprehensive Benchmark for Federated Graph Learning](https://mlsanity.com/paper/2408.16288/) (2024)
- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Threats and Defenses in Federated Learning Life Cycle: A Comprehensive Survey and Challenges](https://mlsanity.com/paper/2407.06754/) (2024)
- [Deep Efficient Private Neighbor Generation for Subgraph Federated Learning](https://mlsanity.com/paper/2401.04336/) (2024)
- [Differentially Private Model Compression](https://mlsanity.com/paper/2206.01838/) (2022)
- [FedMood: Federated Learning on Mobile Health Data for Mood Detection](https://mlsanity.com/paper/2102.09342/) (2021)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [Decentralized Learning with Multi-Headed Distillation](https://mlsanity.com/paper/2211.15774/) (2022)
- [Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacy](https://mlsanity.com/paper/2305.00873/) (2023)
- [Efficient Federated Learning with Enhanced Privacy via Lottery Ticket Pruning in Edge Computing](https://mlsanity.com/paper/2305.01387/) (2023)
- [Federated and Transfer Learning: A Survey on Adversaries and Defense Mechanisms](https://mlsanity.com/paper/2207.02337/) (2022)
- [FedAUXfdp: Differentially Private One-Shot Federated Distillation](https://mlsanity.com/paper/2205.14960/) (2022)
- [Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectives](https://mlsanity.com/paper/2307.02140/) (2023)

## Abstract (arXiv)

Conventional federated learning directly averages model weights, which is only possible for collaboration between models with homogeneous architectures. Sharing prediction instead of weight removes this obstacle and eliminates the risk of white-box inference attacks in conventional federated learning. However, the predictions from local models are sensitive and would leak training data privacy to the public. To address this issue, one naive approach is adding the differentially private random noise to the predictions, which however brings a substantial trade-off between privacy budget and model performance. In this paper, we propose a novel framework called FEDMD-NFDP, which applies a Noise-Free Differential Privacy (NFDP) mechanism into a federated model distillation framework. Our extensive experimental results on various datasets validate that FEDMD-NFDP can deliver not only comparable utility and communication efficiency but also provide a noise-free differential privacy guarantee. We also demonstrate the feasibility of our FEDMD-NFDP by considering both IID and non-IID setting, heterogeneous model architectures, and unlabelled public datasets from a different distribution.