# How and When Adversarial Robustness Transfers in Knowledge Distillation?

arXiv: https://arxiv.org/abs/2110.12072 · HTML: https://mlsanity.com/paper/2110.12072/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 22 Oct 2021.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Vision Transformers are Robust Learners](https://mlsanity.com/paper/2105.07581/) (2021)
- [Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning](https://mlsanity.com/paper/2003.12862/) (2020)
- [Adversarial Robustness through Local Linearization](https://mlsanity.com/paper/1907.02610/) (2019)
- [Adversarially Robust Distillation](https://mlsanity.com/paper/1905.09747/) (2019)
- [Adversarially robust transfer learning](https://mlsanity.com/paper/1905.08232/) (2019)
- [ALP-KD: Attention-Based Layer Projection for Knowledge Distillation](https://mlsanity.com/paper/2012.14022/) (2020)
- [Provable Robustness of ReLU networks via Maximization of Linear Regions](https://mlsanity.com/paper/1810.07481/) (2018)
- [Private Model Compression via Knowledge Distillation](https://mlsanity.com/paper/1811.05072/) (2018)
- [What it Thinks is Important is Important: Robustness Transfers through Input Gradients](https://mlsanity.com/paper/1912.05699/) (2019)
- [On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learning](https://mlsanity.com/paper/2102.10454/) (2021)
- [Towards Robust, Locally Linear Deep Networks](https://mlsanity.com/paper/1907.03207/) (2019)
- [Locally Linear Attributes of ReLU Neural Networks](https://mlsanity.com/paper/2012.01940/) (2020)

## Built on by (most cited first, 16 of 16)

- [On the benefits of knowledge distillation for adversarial robustness](https://mlsanity.com/paper/2203.07159/) (2022)
- [Do Perceptually Aligned Gradients Imply Adversarial Robustness?](https://mlsanity.com/paper/2207.11378/) (2022)
- [Squeeze Training for Adversarial Robustness](https://mlsanity.com/paper/2205.11156/) (2022)
- [Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation](https://mlsanity.com/paper/2305.13067/) (2023)
- [Enhancing Adversarial Robustness via Uncertainty-Aware Distributional Adversarial Training](https://mlsanity.com/paper/2411.02871/) (2024)
- [Robust feature knowledge distillation for enhanced performance of lightweight crack segmentation models](https://mlsanity.com/paper/2404.06258/) (2024)
- [Knowledge Distillation Must Account for What It Loses](https://mlsanity.com/paper/2604.25110/) (2026)
- [Maximum Likelihood Distillation for Robust Modulation Classification](https://mlsanity.com/paper/2211.00748/) (2022)
- [Releasing Inequality Phenomenon in ℓ∞-norm Adversarial Training via Input Gradient Distillation](https://mlsanity.com/paper/2305.09305/) (2023)
- [AdvFunMatch: When Consistent Teaching Meets Adversarial Robustness](https://mlsanity.com/paper/2305.14700/) (2023)
- [A Theoretical Perspective on Subnetwork Contributions to Adversarial Robustness](https://mlsanity.com/paper/2307.03803/) (2023)
- [DAD++: Improved Data-free Test Time Adversarial Defense](https://mlsanity.com/paper/2309.05132/) (2023)
- [Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graph](https://mlsanity.com/paper/2405.08547/) (2024)
- [ProARD: progressive adversarial robustness distillation: provide wide range of robust students](https://mlsanity.com/paper/2506.07666/) (2025)
- [Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation](https://mlsanity.com/paper/2512.10275/) (2025)
- [Toward Understanding Adversarial Distillation: Why Robust Teachers Fail](https://mlsanity.com/paper/2605.21999/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) has been widely used in teacher-student training, with applications to model compression in resource-constrained deep learning. Current works mainly focus on preserving the accuracy of the teacher model. However, other important model properties, such as adversarial robustness, can be lost during distillation. This paper studies how and when the adversarial robustness can be transferred from a teacher model to a student model in KD. We show that standard KD training fails to preserve adversarial robustness, and we propose KD with input gradient alignment (KDIGA) for remedy. Under certain assumptions, we prove that the student model using our proposed KDIGA can achieve at least the same certified robustness as the teacher model. Our experiments of KD contain a diverse set of teacher and student models with varying network architectures and sizes evaluated on ImageNet and CIFAR-10 datasets, including residual neural networks (ResNets) and vision transformers (ViTs). Our comprehensive analysis shows several novel insights that (1) With KDIGA, students can preserve or even exceed the adversarial robustness of the teacher model, even when their models have fundamentally different architectures; (2) KDIGA enables robustness to transfer to pre-trained students, such as KD from an adversarially trained ResNet to a pre-trained ViT, without loss of clean accuracy; and (3) Our derived local linearity bounds for characterizing adversarial robustness in KD are consistent with the empirical results.