mlsanityData to 14 Oct 2026

Unveiling Client Privacy Leakage from Public Dataset Usage in Federated Distillation

arXiv 2502.08001 ↗First posted 11 Feb 2025cs.CR cs.LGcited by 0

On the map: LLM safety and security → Federated unlearning and poisoning

Abstract (arXiv)

Federated Distillation (FD) has emerged as a popular federated training framework, enabling clients to collaboratively train models without sharing private data. Public Dataset-Assisted Federated Distillation (PDA-FD), which leverages public datasets for knowledge sharing, has become widely adopted. Although PDA-FD enhances privacy compared to traditional Federated Learning, we demonstrate that the use of public datasets still poses significant privacy risks to clients' private training data. This paper presents the first comprehensive privacy analysis of PDA-FD in presence of an honest-but-curious server. We show that the server can exploit clients' inference results on public datasets to extract two critical types of private information: label distributions and membership information of the private training dataset. To quantify these vulnerabilities, we introduce two novel attacks specifically designed for the PDA-FD setting: a label distribution inference attack and innovative membership inference methods based on Likelihood Ratio Attack (LiRA). Through extensive evaluation of three representative PDA-FD frameworks (FedMD, DS-FL, and Cronus), our attacks achieve state-of-the-art performance, with label distribution attacks reaching minimal KL-divergence and membership inference attacks maintaining high True Positive Rates under low False Positive Rate constraints. Our findings reveal significant privacy risks in current PDA-FD frameworks and emphasize the need for more robust privacy protection mechanisms in collaborative learning systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated unlearning and poisoning, one of 12 directions in LLM safety and security. Builds on 22 of its 43 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

22 of 43 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  6. 2016Deep Learning with Differential Privacycited by 2,728
  7. 2021Membership Inference Attacks From First Principlescited by 878
  8. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 520
  9. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  10. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 248
  11. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  12. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 147
10 more
  1. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  2. 2019Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learningcited by 122
  3. 2022Membership Inference Attacks by Exploiting Loss Trajectorycited by 101
  4. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  5. 2017Being Robust (in High Dimensions) Can Be Practicalcited by 83
  6. 2021Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecturecited by 67
  7. 2022Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queriescited by 40
  8. 2023Overconfidence is a Dangerous Thing: Mitigating Membership Inference Attacks by Enforcing Less Confident Predictioncited by 29
  9. 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillationcited by 25
  10. 2021User-Level Label Leakage from Gradients in Federated Learningcited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Secure Federated Data Distillation
  2. 2024Privacy-Preserving Federated Learning via Dataset Distillation
  3. 2023Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher
  4. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation
  5. 2025Federated One-Shot Learning with Data Privacy and Objective-Hiding
  6. 2023Privacy and Accuracy Implications of Model Complexity and Integration in Heterogeneous Federated Learning
  7. 2024Privacy in Federated Learning
  8. 2024Membership Information Leakage in Federated Contrastive Learning