Data to 7 Oct 2026

Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks

arXiv 1511.04508 ↗First posted 14 Nov 2015cs.CR cs.LG cs.NEcited by 675

Not on the map. Why?

Abstract (arXiv)

Deep learning algorithms have been shown to perform extremely well on many classical machine learning problems. However, recent studies have shown that deep learning, like other machine learning techniques, is vulnerable to adversarial samples: inputs crafted to force a deep neural network (DNN) to provide adversary-selected outputs. Such attacks can seriously undermine the security of the system supported by the DNN, sometimes with devastating consequences. For example, autonomous vehicles can be crashed, illicit or illegal content can bypass content filters, or biometric authentication systems can be manipulated to allow improper access. In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs. We analytically investigate the generalizability and robustness properties granted by the use of defensive distillation when training DNNs. We also empirically study the effectiveness of our defense mechanisms on two DNNs placed in adversarial settings. The study shows that defensive distillation can reduce effectiveness of sample creation from 95% to less than 0.5% on a studied DNN. Such dramatic gains can be explained by the fact that distillation leads gradients used in adversarial sample creation to be reduced by a factor of 10^30. We also find that distillation increases the average minimum number of features that need to be modified to create adversarial samples by about 800% on one of the DNNs we tested.

Key facts. Cited by 675 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 217; 2022: 159; 2023: 128; 2024: 91; 2025: 63; 2026: 17.

675 papers.

Sort:

By area:

2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 201
2021Data Augmentation Can Improve Robustnesscited by 193
2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
2021Does Knowledge Distillation Really Work?cited by 187
2021Towards Understanding Knowledge Distillationcited by 158
2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
2021Graph Unlearningcited by 129
2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
2021On the Convergence and Robustness of Adversarial Trainingcited by 127
2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116

38 more

2021Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspectivecited by 116
2021Stable Neural ODE with Lyapunov-Stable Equilibrium Points for Defending Against Adversarial Attackscited by 91
2021ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Modelscited by 79
2021RobustART: Benchmarking Robustness on Architecture Design and Training Techniquescited by 78
2021Improving Adversarial Robustness via Channel-wise Activation Suppressingcited by 76
2022Federated Unlearning with Knowledge Distillationcited by 70
2023How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defensescited by 65
2022Evaluating the Adversarial Robustness of Adaptive Test-time Defensescited by 64
2021PatchCleanser: Certifiably Robust Defense against Adversarial Patches for Any Image Classifiercited by 63
2022Factorizing Knowledge in Neural Networkscited by 59
2021Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Trainingcited by 58
2021Privacy-Preserving Machine Learning: Methods, Challenges and Directionscited by 58
2021Socially Responsible AI Algorithms: Issues, Purposes, and Challengescited by 56
2021Reliable Adversarial Distillation with Unreliable Teacherscited by 55
2022Robust Semantic Communications with Masked VQ-VAE Enabled Codebookcited by 53
2023CFA: Class-wise Calibrated Fair Adversarial Trainingcited by 53
2021Adversarial Example Detection for DNN Models: A Review and Experimental Comparisoncited by 51
2021Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraintscited by 50
2021A Fast Knowledge Distillation Framework for Visual Recognitioncited by 50
2021Model Stealing Attacks Against Inductive Graph Neural Networkscited by 50
2022AdvDO: Realistic Adversarial Attacks for Trajectory Predictioncited by 48
2023Adversarial attacks and defenses in explainable artificial intelligence: A surveycited by 47
2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46
2021Survey: Leakage and Privacy at Inference Timecited by 44
2023How Smooth Is Attention?cited by 44
2021Reveal of Vision Transformers Robustness against Adversarial Attackscited by 43
2022Make Some Noise: Reliable and Efficient Single-Step Adversarial Trainingcited by 42
2022Enhancing Adversarial Training with Second-Order Statistics of Weightscited by 42
2021A Comprehensive Evaluation Framework for Deep Model Robustnesscited by 41
2021An Overview of Backdoor Attacks Against Deep Neural Networks and Possible Defencescited by 41
2021Subspace Adversarial Trainingcited by 41
2021Co-advise: Cross Inductive Bias Distillationcited by 40
2021On the Certified Robustness for Ensemble Models and Beyondcited by 40
2021Adversarial Robustness with Semi-Infinite Constrained Learningcited by 39
2021Pyramid Adversarial Training Improves ViT Performancecited by 39
2022Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translationcited by 39
2021Sinkhorn Distributionally Robust Optimizationcited by 38
2022Self-Ensemble Adversarial Training for Improved Robustnesscited by 38

50 most cited of 675 All 675 (JSON)