mlsanityData to 14 Oct 2026

Distilling the Undistillable: Learning from a Nasty Teacher

arXiv 2210.11728 ↗First posted 21 Oct 2022cs.CVcited by 4

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

The inadvertent stealing of private/sensitive information using Knowledge Distillation (KD) has been getting significant attention recently and has guided subsequent defense efforts considering its critical nature. Recent work Nasty Teacher proposed to develop teachers which can not be distilled or imitated by models attacking it. However, the promise of confidentiality offered by a nasty teacher is not well studied, and as a further step to strengthen against such loopholes, we attempt to bypass its defense and steal (or extract) information in its presence successfully. Specifically, we analyze Nasty Teacher from two different directions and subsequently leverage them carefully to develop simple yet efficient methodologies, named as HTC and SCM, which increase the learning from Nasty Teacher by upto 68.63% on standard datasets. Additionally, we also explore an improvised defense method based on our insights of stealing. Our detailed set of experiments and ablations on diverse models/settings demonstrate the efficacy of our approach.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 33 of its 42 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

33 of 42 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  10. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  11. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  12. 2019Relational Knowledge Distillationcited by 889
21 more
  1. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  2. 2018Born Again Neural Networkscited by 546
  3. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  4. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  5. 2017Embedding Watermarks into Deep Neural Networkscited by 248
  6. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 246
  7. 2019Data-Free Learning of Student Networkscited by 204
  8. 2018PRADA: Protecting against DNN Model Stealing Attackscited by 165
  9. 2019Adversarially Robust Distillationcited by 125
  10. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 116
  11. 2019[Extended version] Rethinking Deep Neural Network Ownership Verification: Embedding Passports to Defeat Ambiguity Attackscited by 88
  12. 2019Defending Against Model Stealing Attacks with Adaptive Misinformationcited by 57
  13. 2020Model Watermarking for Image Processing Networkscited by 56
  14. 2022Label-Only Model Inversion Attacks via Boundary Repulsioncited by 56
  15. 2020Passport-aware Normalization for Deep Model Protectioncited by 43
  16. 2020Self-supervised Knowledge Distillation for Few-shot Learningcited by 41
  17. 2017Simple And Efficient Architecture Search for Convolutional Neural Networkscited by 40
  18. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
  19. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  20. 2020Black-Box Face Recovery from Identity Featurescited by 20
  21. 2019GAMIN: An Adversarial Approach to Black-Box Model Inversioncited by 19

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 1.

4 papers.

  1. 2023Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillationcited by 29
  2. 2025Towards Undistillable Models by Minimizing Conditional Mutual Informationcited by 3
  3. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2
  4. 2023Lightweight Self-Knowledge Distillation with Multi-source Information Fusioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Revisiting Data-Free Knowledge Distillation with Poisoned Teachers
  2. 2025When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need
  3. 2022Swing Distillation: A Privacy-Preserving Knowledge Distillation Framework
  4. 2025Pay Attention to the Triggers: Constructing Backdoors That Survive Distillation
  5. 2023Learning the Wrong Lessons: Inserting Trojans During Knowledge Distillation
  6. 2023Students Parrot Their Teachers: Membership Inference on Model Distillation
  7. 2021Safe Distillation Box
  8. 2025Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack