# Locally Differentially Private Distributed Deep Learning via Knowledge Distillation

arXiv: https://arxiv.org/abs/2202.02971 · HTML: https://mlsanity.com/paper/2202.02971/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 55 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 25 of 25)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Differentially Private Federated Learning: A Client Level Perspective](https://mlsanity.com/paper/1712.07557/) (2017)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response](https://mlsanity.com/paper/1407.6981/) (2014)
- [Neural Trojans](https://mlsanity.com/paper/1710.00942/) (2017)
- [Collecting Telemetry Data Privately](https://mlsanity.com/paper/1712.01524/) (2017)
- [Minimax Optimal Procedures for Locally Private Estimation](https://mlsanity.com/paper/1604.02390/) (2016)
- [Discrete Distribution Estimation under Local Privacy](https://mlsanity.com/paper/1602.07387/) (2016)
- [Collecting and Analyzing Multidimensional Data with Local Differential Privacy](https://mlsanity.com/paper/1907.00782/) (2019)
- [Local, Private, Efficient Protocols for Succinct Histograms](https://mlsanity.com/paper/1504.04686/) (2015)
- [Optimal Schemes for Discrete Distribution Estimation under Locally Differential Privacy](https://mlsanity.com/paper/1702.00610/) (2017)
- [Clustering Algorithms for the Centralized and Local Models](https://mlsanity.com/paper/1707.04766/) (2017)
- [AutoGAN-based Dimension Reduction for Privacy Preservation](https://mlsanity.com/paper/1902.10799/) (2019)
- [Solo or Ensemble? Choosing a CNN Architecture for Melanoma Classification](https://mlsanity.com/paper/1904.12724/) (2019)
- [DynaMo: Dynamic Community Detection by Incrementally Maximizing Modularity](https://mlsanity.com/paper/1709.08350/) (2017)
- [Locally Differentially Private Naive Bayes Classification](https://mlsanity.com/paper/1905.01039/) (2019)
- [Enhanced PeerHunter: Detecting Peer-to-peer Botnets through Network-Flow Level Community Behavior Analysis](https://mlsanity.com/paper/1802.08386/) (2018)
- [PeerHunter: Detecting Peer-to-Peer Botnets through Community Behavior Analysis](https://mlsanity.com/paper/1709.06440/) (2017)

## Built on by (most cited first, 1 of 1)

- [Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer](https://mlsanity.com/paper/2208.01161/) (2022)

## Abstract (arXiv)

Deep learning often requires a large amount of data. In real-world applications, e.g., healthcare applications, the data collected by a single organization (e.g., hospital) is often limited, and the majority of massive and diverse data is often segregated across multiple organizations. As such, it motivates the researchers to conduct distributed deep learning, where the data user would like to build DL models using the data segregated across multiple different data owners. However, this could lead to severe privacy concerns due to the sensitive nature of the data, thus the data owners would be hesitant and reluctant to participate. We propose LDP-DL, a privacy-preserving distributed deep learning framework via local differential privacy and knowledge distillation, where each data owner learns a teacher model using its own (local) private dataset, and the data user learns a student model to mimic the output of the ensemble of the teacher models. In the experimental evaluation, a comprehensive comparison has been made among our proposed approach (i.e., LDP-DL), DP-SGD, PATE and DP-FL, using three popular deep learning benchmark datasets (i.e., CIFAR10, MNIST and FashionMNIST). The experimental results show that LDP-DL consistently outperforms the other competitors in terms of privacy budget and model accuracy.