mlsanityData to 14 Oct 2026

Knowledge Cross-Distillation for Membership Privacy

arXiv 2111.01363 ↗First posted 2 Nov 2021cs.CR cs.LGcited by 6

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

A membership inference attack (MIA) poses privacy risks for the training data of a machine learning model. With an MIA, an attacker guesses if the target data are a member of the training dataset. The state-of-the-art defense against MIAs, distillation for membership privacy (DMP), requires not only private data for protection but a large amount of unlabeled public data. However, in certain privacy-sensitive domains, such as medicine and finance, the availability of public data is not guaranteed. Moreover, a trivial method for generating public data by using generative adversarial networks significantly decreases the model accuracy, as reported by the authors of DMP. To overcome this problem, we propose a novel defense against MIAs that uses knowledge distillation without requiring public data. Our experiments show that the privacy protection and accuracy of our defense are comparable to those of DMP for the benchmark tabular datasets used in MIA research, Purchase100 and Texas100, and our defense has a much better privacy-utility trade-off than those of the existing defenses that also do not use public data for the image dataset CIFAR10.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 45 of its 80 references that are in the map. First posted on arXiv 2 Nov 2021.

Builds on

45 of 80 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture15 Oct
  2. 2021Membership Inference Attacks on Machine Learning: A Survey14 Mar
  3. 2021On the (In)Feasibility of Attribute Inference Attacks on Machine Learning Models12 Mar
  4. 2021Practical Blind Membership Inference Attack via Differential Comparisons5 Jan

Other references · 41

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2013Auto-Encoding Variational Bayescited by 4,053
  6. 2016Improved Techniques for Training GANscited by 2,970
  7. 2016Deep Learning with Differential Privacycited by 2,728
  8. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  9. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  10. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  11. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  12. 2016Stealing Machine Learning Models via Prediction APIscited by 619
29 more
  1. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 520
  2. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 498
  3. 2020Label-Only Membership Inference Attackscited by 388
  4. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  5. 2019HopSkipJumpAttack: A Query-Efficient Decision-Based Attackcited by 333
  6. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 292
  7. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 271
  8. 2018Scalable Private Learning with PATEcited by 269
  9. 2020Information Leakage in Embedding Modelscited by 203
  10. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 197
  11. 2018Machine Learning with Membership Privacy using Adversarial Regularizationcited by 186
  12. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 175
  13. 2017Machine Learning Models that Remember Too Muchcited by 169
  14. 2019Privacy Risks of Securing Machine Learning Models against Adversarial Examplescited by 141
  15. 2013Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifierscited by 130
  16. 2019Reconstruction and Membership Inference Attacks against Generative Modelscited by 122
  17. 2018Stealing Hyperparameters in Machine Learningcited by 118
  18. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 107
  19. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  20. 2018Understanding Membership Inferences on Well-Generalized Learning Modelscited by 90
  21. 2019Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?cited by 71
  22. 2017Knock Knock, Who's There? Membership Inference on Aggregate Location Datacited by 70
  23. 2018Algorithms that Remember: Model Inversion Attacks and Data Protection Lawcited by 55
  24. 2019Segmentations-Leak: Membership Inference Attacks and Defenses in Semantic Image Segmentationcited by 32
  25. 2018I Know What You See: Power Side-Channel Attack on Convolutional Neural Network Acceleratorscited by 27
  26. 2018Performing Co-Membership Attacks Against Deep Generative Modelscited by 26
  27. 2019Alleviating Privacy Attacks via Causal Learningcited by 19
  28. 2019Effects of Differential Privacy and Data Skewness on Membership Inference Vulnerabilitycited by 17
  29. 2019Characterizing Membership Privacy in Stochastic Gradient Langevin Dynamicscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 1; 2026: 1.

6 papers.

  1. 2023Overconfidence is a Dangerous Thing: Mitigating Membership Inference Attacks by Enforcing Less Confident Predictioncited by 29
  2. 2022NeuGuard: Lightweight Neuron-Guided Defense against Membership Inference Attackscited by 3
  3. 2024Model Mimic Attack: Knowledge Distillation for Provably Transferable Adversarial Examplescited by 2
  4. 2025On Membership Inference Attacks in Knowledge Distillationcited by 1
  5. 2024Survey of Privacy Threats and Countermeasures in Federated Learningcited by 0
  6. 2026Privacy-Preserving Model Transcription with Differentially Private Synthetic Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Students Parrot Their Teachers: Membership Inference on Model Distillation
  2. 2023Diffence: Fencing Membership Privacy With Diffusion Models
  3. 2024Bayes-Nash Generative Privacy Against Membership Inference Attacks
  4. 2022Membership Privacy Protection for Image Translation Models via Adversarial Knowledge Distillation
  5. 2026A Critical Review on the Effectiveness and Privacy Threats of Membership Inference Attacks
  6. 2025Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble
  7. 2025Privacy Leaks by Adversaries: Adversarial Iterations for Membership Inference Attack
  8. 2021On the Privacy Risks of Deploying Recurrent Neural Networks in Machine Learning Models