mlsanityData to 7 Oct 2026

LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training

arXiv 2111.02331 ↗First posted 3 Nov 2021cs.CV cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Adversarial training is a widely adopted strategy to bolster the robustness of neural network models against adversarial attacks. This paper revisits the fundamental assumptions underlying image classification and suggests that representing data as one-hot labels is a key factor that leads to vulnerabilities. However, in real-world datasets, data ambiguity often arises, with samples exhibiting characteristics of multiple classes, rendering one-hot label representations imprecise. To address this, we introduce a novel approach, Low-Temperature Distillation (LTD), designed to refine label representations. Unlike previous approaches, LTD incorporates a relatively low temperature in the teacher model, while maintaining a fixed temperature for the student model during both training and inference. This strategy not only refines assumptions about data distribution but also strengthens model robustness and avoids the gradient masking problem commonly encountered in defensive distillation. Experimental results demonstrate the efficacy of the proposed method when combined with existing frameworks, achieving robust accuracy rates of 58.19%, 31.13%, and 42.08% on the CIFAR-10, CIFAR-100, and ImageNet datasets, respectively, without the need for additional data.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 3 Nov 2021.

Builds on

43 of 61 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2016Wide Residual Networkscited by 3,057
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2016Adversarial examples in the physical worldcited by 1,762
  9. 2020Knowledge Distillation: A Surveycited by 1,578
  10. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  11. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  12. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
31 more
  1. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  2. 2019When Does Label Smoothing Help?cited by 985
  3. 2016Adversarial Machine Learning at Scalecited by 934
  4. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  5. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  6. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  7. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  8. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  9. 2019Adversarial Training for Free!cited by 638
  10. 2020Overfitting in adversarially robust deep learningcited by 504
  11. 2019On Evaluating Adversarial Robustnesscited by 442
  12. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  13. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 401
  14. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  15. 2020Are we done with ImageNet?cited by 329
  16. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  17. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 275
  18. 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 242
  19. 2020Bag of Tricks for Adversarial Trainingcited by 199
  20. 2019Image Captioning: Transforming Objects into Wordscited by 171
  21. 2019Are Labels Required for Improving Adversarial Robustness?cited by 162
  22. 2018Deep Learning for Sentiment Analysis : A Surveycited by 139
  23. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  24. 2020Boosting Adversarial Training with Hypersphere Embeddingcited by 97
  25. 2020Learnable Boundary Guided Adversarial Trainingcited by 93
  26. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
  27. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 77
  28. 2020Adversarial Vertex Mixup: Toward Better Adversarially Robust Generalizationcited by 73
  29. 2020The Translucent Patch: A Physical and Universal Attack on Object Detectorscited by 62
  30. 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 55
  31. 2020Towards Robust Pattern Recognition: A Reviewcited by 31

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 3; 2025: 2.

13 papers.

Sort:

  1. 2023How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defensescited by 65
  2. 2023Overload: Latency Attacks on Object Detection for Edge Devicescited by 13
  3. 2024Machine Learning Robustness: A Primercited by 8
  4. 2023Annealing Self-Distillation Rectification Improves Adversarial Trainingcited by 7
  5. 2023Indirect Gradient Matching for Adversarial Robust Distillationcited by 5
  6. 2024Dynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledgecited by 5
  7. 2023IRAD: Implicit Representation-driven Image Resampling against Adversarial Attackscited by 1
  8. 2023Topology-preserving Adversarial Training for Alleviating Natural Accuracy Degradationcited by 1
  9. 2025Democratic Training Against Universal Adversarial Perturbationscited by 1
  10. 2023Denoising Autoencoder-based Defensive Distillation as an Adversarial Robustness Algorithmcited by 0
  11. 2023Revisiting and Advancing Adversarial Training Through A Simple Baselinecited by 0
  12. 2024On adversarial training and the 1 Nearest Neighbor classifiercited by 0
1 more
  1. 2025Revisiting the Relationship between Adversarial and Clean Training: Why Clean Training Can Make Adversarial Training Bettercited by 0