mlsanityData to 7 Oct 2026

Extreme Miscalibration and the Illusion of Adversarial Robustness

arXiv 2402.17509 ↗First posted 27 Feb 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Deep learning-based Natural Language Processing (NLP) models are vulnerable to adversarial attacks, where small perturbations can cause a model to misclassify. Adversarial Training (AT) is often used to increase model robustness. However, we have discovered an intriguing phenomenon: deliberately or accidentally miscalibrating models masks gradients in a way that interferes with adversarial attack search methods, giving rise to an apparent increase in robustness. We show that this observed gain in robustness is an illusion of robustness (IOR), and demonstrate how an adversary can perform various forms of test-time temperature calibration to nullify the aforementioned interference and allow the adversarial attack to find adversarial examples. Hence, we urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations to ensure that any observed gains are genuine. Finally, we show how the temperature can be scaled during textittraining to improve genuine robustness.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

34 of 61 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  11. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  12. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
22 more
  1. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  2. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  3. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  4. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  5. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  6. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  7. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  8. 2021Towards Improving Adversarial Training of NLP Modelscited by 79
  9. 2021Towards Robustness Against Natural Language Word Substitutionscited by 68
  10. 2020It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbationscited by 64
  11. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  12. 2021Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitutioncited by 55
  13. 2020Temperature check: theory and practice for training models with softmax-cross-entropy lossescited by 31
  14. 2020Defense against Adversarial Attacks in NLP via Dirichlet Neighborhood Ensemblecited by 26
  15. 2021Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglotscited by 23
  16. 2022A Survey of Adversarial Defences and Robustness in NLPcited by 22
  17. 2022Textual Manifold-based Defense Against Natural Language Adversarial Examplescited by 19
  18. 2023Adversarial Training Should Be Cast as a Non-Zero-Sum Gamecited by 13
  19. 2020TAVAT: Token-Aware Virtual Adversarial Training for Language Understandingcited by 8
  20. 2021Adversarial Training with Contrastive Learning in NLPcited by 6
  21. 2018Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attackscited by 4
  22. 2023Sample Attackability in Natural Language Adversarial Attackscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 10
  2. 2025Confidence Elicitation: A New Attack Vector for Large Language Modelscited by 4
  3. 2024Standard-Deviation-Inspired Regularization for Improving Adversarial Robustnesscited by 0
  4. 2025Breaking the Adversarial Robustness-Performance Trade-off in Text Classification via Manifold Purificationcited by 0