# Adversarial Contrastive Distillation with Adaptive Denoising

arXiv: https://arxiv.org/abs/2302.08764 · HTML: https://mlsanity.com/paper/2302.08764/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 17 Feb 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples](https://mlsanity.com/paper/2010.03593/) (2020)
- [Adversarially Robust Distillation](https://mlsanity.com/paper/1905.09747/) (2019)
- [Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better](https://mlsanity.com/paper/2108.07969/) (2021)
- [CMUA-Watermark: A Cross-Model Universal Adversarial Watermark for Combating Deepfakes](https://mlsanity.com/paper/2105.10872/) (2021)
- [Reliable Adversarial Distillation with Unreliable Teachers](https://mlsanity.com/paper/2106.04928/) (2021)
- [Towards Practical Certifiable Patch Defense with Vision Transformer](https://mlsanity.com/paper/2203.08519/) (2022)
- [Boosting the Transferability of Adversarial Attacks with Global Momentum Initialization](https://mlsanity.com/paper/2211.11236/) (2022)
- [Learning Appearance-motion Normality for Video Anomaly Detection](https://mlsanity.com/paper/2207.13361/) (2022)
- [Efficient universal shuffle attack for visual object tracking](https://mlsanity.com/paper/2203.06898/) (2022)

## Built on by (most cited first, 9 of 9)

- [AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception](https://mlsanity.com/paper/2307.13933/) (2023)
- [Context De-confounded Emotion Recognition](https://mlsanity.com/paper/2303.11921/) (2023)
- [Robust Emotion Recognition in Context Debiasing](https://mlsanity.com/paper/2403.05963/) (2024)
- [Efficient Decision-based Black-box Patch Attacks on Video Recognition](https://mlsanity.com/paper/2303.11917/) (2023)
- [Improving Generalization in Visual Reinforcement Learning via Conflict-aware Gradient Agreement Augmentation](https://mlsanity.com/paper/2308.01194/) (2023)
- [De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts](https://mlsanity.com/paper/2403.19539/) (2024)
- [Sampling to Distill: Knowledge Transfer from Open-World Data](https://mlsanity.com/paper/2307.16601/) (2023)
- [Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation](https://mlsanity.com/paper/2303.11611/) (2023)
- [Self-Cooperation Knowledge Distillation for Novel Class Discovery](https://mlsanity.com/paper/2407.01930/) (2024)

## Abstract (arXiv)

Adversarial Robustness Distillation (ARD) is a novel method to boost the robustness of small models. Unlike general adversarial training, its robust knowledge transfer can be less easily restricted by the model capacity. However, the teacher model that provides the robustness of knowledge does not always make correct predictions, interfering with the student's robust performances. Besides, in the previous ARD methods, the robustness comes entirely from one-to-one imitation, ignoring the relationship between examples. To this end, we propose a novel structured ARD method called Contrastive Relationship DeNoise Distillation (CRDND). We design an adaptive compensation module to model the instability of the teacher. Moreover, we utilize the contrastive relationship to explore implicit robustness knowledge among multiple examples. Experimental results on multiple attack benchmarks show CRDND can transfer robust knowledge efficiently and achieves state-of-the-art performances.