mlsanityData to 7 Oct 2026

AdvFunMatch: When Consistent Teaching Meets Adversarial Robustness

arXiv 2305.14700 ↗First posted 24 May 2023cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Consistent teaching is an effective paradigm for implementing knowledge distillation (KD), where both student and teacher models receive identical inputs, and KD is treated as a function matching task (FunMatch). However, one limitation of FunMatch is that it does not account for the transfer of adversarial robustness, a model's resistance to adversarial attacks. To tackle this problem, we propose a simple but effective strategy called Adversarial Function Matching (AdvFunMatch), which aims to match distributions for all data points within the ℓp-norm ball of the training data, in accordance with consistent teaching. Formulated as a min-max optimization problem, AdvFunMatch identifies the worst-case instances that maximizes the KL-divergence between teacher and student model outputs, which we refer to as "mismatched examples," and then matches the outputs on these mismatched examples. Our experimental results show that AdvFunMatch effectively produces student models with both high clean accuracy and robustness. Furthermore, we reveal that strong data augmentations (e.g., AutoAugment) are beneficial in AdvFunMatch, whereas prior works have found them less effective in adversarial training. Code is available at urlhttps://gitee.com/zihui998/adv-fun-match.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 49 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

41 of 49 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  9. 2016Wide Residual Networkscited by 3,057
  10. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
29 more
  1. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  4. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  6. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  7. 2019Relational Knowledge Distillationcited by 873
  8. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  9. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  10. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  11. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  12. 2020Overfitting in adversarially robust deep learningcited by 504
  13. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  14. 2020Understanding and Improving Fast Adversarial Trainingcited by 213
  15. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
  16. 2021Data Augmentation Can Improve Robustnesscited by 193
  17. 2021Does Knowledge Distillation Really Work?cited by 187
  18. 2019Adversarially Robust Distillationcited by 124
  19. 2021Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?cited by 113
  20. 2019Adversarial Robustness vs Model Compression, or Both?cited by 98
  21. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 77
  22. 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 55
  23. 2022Efficient and Effective Augmentation Strategy for Adversarial Trainingcited by 43
  24. 2023Data Augmentation Alone Can Improve Adversarial Trainingcited by 43
  25. 2020A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNscited by 37
  26. 2020What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical Perspectivecited by 35
  27. 2019What it Thinks is Important is Important: Robustness Transfers through Input Gradientscited by 34
  28. 2022On the benefits of knowledge distillation for adversarial robustnesscited by 17
  29. 2021How and When Adversarial Robustness Transfers in Knowledge Distillation?cited by 15

Built on by

Not cited yet.