mlsanityData to 14 Oct 2026

Annealing Self-Distillation Rectification Improves Adversarial Training

arXiv 2305.12118 ↗First posted 20 May 2023cs.LG cs.AIcited by 7

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

In standard adversarial training, models are optimized to fit one-hot labels within allowable adversarial perturbation budgets. However, the ignorance of underlying distribution shifts brought by perturbations causes the problem of robust overfitting. To address this issue and enhance adversarial robustness, we analyze the characteristics of robust models and identify that robust models tend to produce smoother and well-calibrated outputs. Based on the observation, we propose a simple yet effective method, Annealing Self-Distillation Rectification (ADR), which generates soft labels as a better guidance mechanism that accurately reflects the distribution shift under attack during adversarial training. By utilizing ADR, we can obtain rectified distributions that significantly improve model robustness without the need for pre-trained models or extensive extra computation. Moreover, our method facilitates seamless plug-and-play integration with other adversarial training techniques by replacing the hard labels in their objectives. We demonstrate the efficacy of ADR through extensive experiments and strong performances across datasets.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 57 of its 69 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

57 of 69 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better18 Aug
  2. 2021Reliable Adversarial Distillation with Unreliable Teachers9 Jun

Other references · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2013Intriguing properties of neural networkscited by 5,155
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  11. 2016Wide Residual Networkscited by 3,070
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
38 more
  1. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  2. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  3. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  4. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  6. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  9. 2019When Does Label Smoothing Help?cited by 988
  10. 2016Adversarial Machine Learning at Scalecited by 936
  11. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
  12. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  13. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  14. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
  15. 2020Overfitting in adversarially robust deep learningcited by 506
  16. 2019On Evaluating Adversarial Robustnesscited by 447
  17. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  18. 2019Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attackcited by 334
  19. 2017Defense against Adversarial Attacks Using High-Level Representation Guided Denoisercited by 329
  20. 2018Adversarially Robust Generalization Requires More Datacited by 318
  21. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 278
  22. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  23. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  24. 2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 204
  25. 2020Bag of Tricks for Adversarial Trainingcited by 200
  26. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 199
  27. 2021Data Augmentation Can Improve Robustnesscited by 194
  28. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 185
  29. 2019Adversarially Robust Distillationcited by 125
  30. 2021Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?cited by 113
  31. 2022LAS-AT: Adversarial Training with Learnable Attack Strategycited by 95
  32. 2020Learnable Boundary Guided Adversarial Trainingcited by 93
  33. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  34. 2021Exploring Memorization in Adversarial Trainingcited by 55
  35. 2022Efficient and Effective Augmentation Strategy for Adversarial Trainingcited by 43
  36. 2023Data Augmentation Alone Can Improve Adversarial Trainingcited by 43
  37. 2018Evaluating and Understanding the Robustness of Adversarial Logit Pairingcited by 34
  38. 2020Improving Calibration through the Relationship with Adversarial Robustnesscited by 26

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 2.

7 papers.

  1. 2025Failure Cases Are Better Learned But Boundary Says Sorry: Facilitating Smooth Perception Change for Accuracy-Robustness Trade-Off in Adversarial Trainingcited by 3
  2. 2024New Paradigm of Adversarial Training: Releasing Accuracy-Robustness Trade-Off via Dummy Classcited by 2
  3. 2024Revisiting Semi-supervised Adversarial Robustness via Noise-aware Online Robust Distillationcited by 1
  4. 2024Adversarial Robustness Overestimation and Instability in TRADEScited by 1
  5. 2025Identifying and Understanding Cross-Class Features in Adversarial Trainingcited by 0
  6. 2026Expanding the Role of Diffusion Models for Robust Classifier Trainingcited by 0
  7. 2026Robust Alignment: Harmonizing Clean Accuracy and Adversarial Robustness in Adversarial Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022ARDIR: Improving Robustness using Knowledge Distillation of Internal Representation
  2. 2025Long-tailed Adversarial Training with Self-Distillation
  3. 2022On the benefits of knowledge distillation for adversarial robustness
  4. 2026Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
  5. 2022Pruning Adversarially Robust Neural Networks without Adversarial Examples
  6. 2025Towards Class-wise Fair Adversarial Training via Anti-Bias Soft Label Distillation
  7. 2022Vanilla Feature Distillation for Improving the Accuracy-Robustness Trade-Off in Adversarial Training
  8. 2022Towards Alternative Techniques for Improving Adversarial Robustness: Analysis of Adversarial Training at a Spectrum of Perturbations