# LDPKiT: Superimposing Remote Queries for Privacy-Preserving Distillation

arXiv: https://arxiv.org/abs/2405.16361 · HTML: https://mlsanity.com/paper/2405.16361/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 25 May 2024.

## Builds on (most cited first, 20 of 20)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response](https://mlsanity.com/paper/1407.6981/) (2014)
- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [Data-Free Model Extraction](https://mlsanity.com/paper/2011.14779/) (2020)
- [Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware](https://mlsanity.com/paper/1806.03287/) (2018)
- [A Survey on Gradient Inversion: Attacks, Defenses and Future Directions](https://mlsanity.com/paper/2206.07284/) (2022)
- [Not Just Privacy: Improving Performance of Private Deep Learning in Mobile Cloud](https://mlsanity.com/paper/1809.03428/) (2018)
- [DeepMarks: A Digital Fingerprinting Framework for Deep Neural Networks](https://mlsanity.com/paper/1804.03648/) (2018)
- [Increasing the Cost of Model Extraction with Calibrated Proof of Work](https://mlsanity.com/paper/2201.09243/) (2022)
- [IDEAL: Query-Efficient Data-Free Learning from Black-box Models](https://mlsanity.com/paper/2205.11158/) (2022)
- [HETAL: Efficient Privacy-preserving Transfer Learning with Homomorphic Encryption](https://mlsanity.com/paper/2403.14111/) (2024)
- [Visual Privacy Auditing with Diffusion Models](https://mlsanity.com/paper/2403.07588/) (2024)

## Built on by (most cited first, 1 of 1)

- [Let the Neurons Die: Exploiting ReLU-Induced Model Degradation](https://mlsanity.com/paper/2609.35528/) (2026)

## Abstract (arXiv)

To protect privacy in regulated domains such as healthcare and finance, model owners may allow only remote API access while keeping both the training data and model parameters private. However, model users performing inference on such remotely hosted models may be required to transmit potentially sensitive inputs, raising privacy concerns. In this work, we present LDPKiT, a framework for non-adversarial, privacy-preserving model distillation that leverages a user's private in-distribution data while bounding privacy leakage. LDPKiT introduces a novel superimposition technique that generates approximately in-distribution samples, enabling effective knowledge transfer under local differential privacy (LDP). Experiments on Fashion-MNIST, SVHN, and PathMNIST demonstrate that LDPKiT consistently improves utility while maintaining privacy, with benefits that become more pronounced at stronger noise levels. For example, on SVHN, LDPKiT achieves nearly the same inference accuracy at ε=1.25 as at ε=2.0, yielding stronger privacy guarantees with less than a 2% accuracy reduction. We further conduct sensitivity analyses to examine the effect of dataset size on performance and provide a systematic analysis of latent space representations, offering intuitive and empirical insights into the accuracy gains of LDPKiT.