Faithful Knowledge Distillation
Not on the map. Why?
Abstract (arXiv)
Knowledge distillation (KD) has received much attention due to its success in compressing networks to allow for their deployment in resource-constrained systems. While the problem of adversarial robustness has been studied before in the KD setting, previous works overlook what we term the relative calibration of the student network with respect to its teacher in terms of soft confidences. In particular, we focus on two crucial questions with regard to a teacher-student pair: (i) do the teacher and student disagree at points close to correctly classified dataset examples, and (ii) is the distilled student as confident as the teacher around dataset examples? These are critical questions when considering the deployment of a smaller student network trained from a robust teacher within a safety-critical setting. To address these questions, we introduce a faithful imitation framework to discuss the relative calibration of confidences and provide empirical and certified methods to evaluate the relative calibration of a student w.r.t. its teacher. Further, to verifiably align the relative calibration incentives of the student to those of its teacher, we introduce faithful distillation. Our experiments on the MNIST, Fashion-MNIST and CIFAR-10 datasets demonstrate the need for such an analysis and the advantages of the increased verifiability of faithful distillation over alternative adversarial distillation methods.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 7 Jun 2023.
Builds on
23 of 30 references.
References · 23
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2020Knowledge Distillation: A Surveycited by 1,578
- 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
- 2017Countering Adversarial Images using Input Transformationscited by 517
- 2019Similarity-Preserving Knowledge Distillationcited by 507
- 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
- 2018Efficient Neural Network Robustness Certification with General Activation Functionscited by 439
11 more
- 2018Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Modelscited by 362
- 2017PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examplescited by 243
- 2017Formal Verification of Piece-Wise Linear Feed-Forward Neural Networkscited by 241
- 2019Adversarially Robust Distillationcited by 124
- 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 100
- 2019NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networkscited by 94
- 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 77
- 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 55
- 2018Private Model Compression via Knowledge Distillationcited by 37
- 2017Cascade Adversarial Machine Learning Regularized with a Unified Embeddingcited by 13
- 2018Robust Student Network Learningcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.