mlsanityData to 7 Oct 2026

Robustness Verification for Transformers

arXiv 2002.06622 ↗First posted 16 Feb 2020cs.LG stat.MLcited by 76

Not on the map. Why?

Abstract (arXiv)

Robustness verification that aims to formally certify the prediction behavior of neural networks has become an important tool for understanding model behavior and obtaining safety guarantees. However, previous methods can usually only handle neural networks with relatively simple architectures. In this paper, we consider the robustness verification problem for Transformers. Transformers have complex self-attention layers that pose many challenges for verification, including cross-nonlinearity and cross-position dependency, which have not been discussed in previous works. We resolve these challenges and develop the first robustness verification algorithm for Transformers. The certified robustness bounds computed by our method are significantly tighter than those by naive Interval Bound Propagation. These bounds also shed light on interpreting Transformers as they consistently reflect the importance of different words in sentiment analysis.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 23; 2023: 14; 2024: 9; 2025: 8; 2026: 8.

76 papers.

Sort:

By area:

  1. 2021Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Robustness Verificationcited by 224
  2. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  3. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  4. 2021On the Adversarial Robustness of Vision Transformerscited by 104
  5. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  6. 2021Certified Robustness to Text Adversarial Attacks by Randomized [MASK]cited by 57
  7. 2021Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNscited by 57
  8. 2021Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitutioncited by 55
  9. 2024Neural Network Verification with Branch-and-Bound for General Nonlinearitiescited by 47
  10. 2022Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagationcited by 45
  11. 2022Adversarial Robustness of Deep Neural Networks: A Survey from a Formal Verification Perspectivecited by 41
  12. 2021On the Sensitivity and Stability of Model Interpretations in NLPcited by 30
38 more
  1. 2021Achieving Model Robustness through Discrete Adversarial Trainingcited by 23
  2. 2023Certified Robustness for Large Language Models with Self-Denoisingcited by 23
  3. 2022A Survey of Adversarial Defences and Robustness in NLPcited by 22
  4. 2021Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Trainingcited by 21
  5. 2022Textual Manifold-based Defense Against Natural Language Adversarial Examplescited by 19
  6. 2023Convex Bounds on the Softmax Function with Applications to Robustness Verificationcited by 19
  7. 2023Robustifying Token Attention for Vision Transformerscited by 19
  8. 2021Certified Robustness to Programmable Transformations in LSTMscited by 18
  9. 2021Blending Anti-Aliasing into Vision Transformercited by 16
  10. 2022On Robust Prefix-Tuning for Text Classificationcited by 15
  11. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 13
  12. 2022Robust Textual Embedding against Word-level Adversarial Attackscited by 12
  13. 2022Adversarial Training for Improving Model Robustness? Look at Both Prediction and Interpretationcited by 12
  14. 2024Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptionscited by 12
  15. 2022LinSyn: Synthesizing Tight Linear Bounds for Arbitrary Neural Network Activation Functionscited by 10
  16. 2022Localized Randomized Smoothing for Collective Robustness Certificationcited by 10
  17. 2023Efficient Error Certification for Physics-Informed Neural Networkscited by 10
  18. 2023Improving the Robustness of Transformer-based Large Language Models with Dynamic Attentioncited by 10
  19. 2021Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluationcited by 9
  20. 2022Robust Natural Language Processing: Recent Advances, Challenges, and Future Directionscited by 9
  21. 2022CAISAR: A platform for Characterizing Artificial Intelligence Safety and Robustnesscited by 9
  22. 2022Impact of Adversarial Training on Robustness and Generalizability of Language Modelscited by 9
  23. 2023Less is More: Understanding Word-level Textual Adversarial Attack via n-gram Frequency Descendcited by 9
  24. 2021Detecting Textual Adversarial Examples through Randomized Substitution and Votecited by 8
  25. 2022Can Transformers Reason in Fragments of Natural Language?cited by 8
  26. 2023TextShield: Beyond Successfully Detecting Adversarial Sentences in Text Classificationcited by 8
  27. 2023ANTONIO: Towards a Systematic Method of Generating NLP Benchmarks for Verificationcited by 6
  28. 2023Rate-Adaptive Coding Mechanism for Semantic Communications With Multi-Modal Datacited by 6
  29. 2022Are Transformers More Robust? Towards Exact Robustness Verification for Transformerscited by 5
  30. 20223DVerifier: Efficient Robustness Verification for 3D Point Cloud Modelscited by 5
  31. 2024Transformer Encoder Satisfiability: Complexity and Impact on Formal Reasoningcited by 5
  32. 2024SoundnessBench: A Soundness Benchmark for Neural Network Verifierscited by 5
  33. 2022Identifying Adversarial Attacks on Text Classifierscited by 4
  34. 2022Disentangled Text Representation Learning with Information-Theoretic Perspective for Adversarial Robustnesscited by 4
  35. 2025Exact Verification of Graph Neural Networks with Incremental Constraint Solvingcited by 4
  36. 2023Robustness Verification for Knowledge-Based Logic of Risky Driving Scenescited by 3
  37. 2022Quantifying Robustness to Adversarial Word Substitutionscited by 2
  38. 2022TCAB: A Large-Scale Text Classification Attack Benchmarkcited by 2

50 most cited of 76 All 76 (JSON)