mlsanityData to 7 Oct 2026

SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization

arXiv 2402.03317 ↗First posted 2 Jan 2024cs.CV cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViTs) are increasingly used in computer vision due to their high performance, but their vulnerability to adversarial attacks is a concern. Existing methods lack a solid theoretical basis, focusing mainly on empirical training adjustments. This study introduces SpecFormer, tailored to fortify ViTs against adversarial attacks, with theoretical underpinnings. We establish local Lipschitz bounds for the self-attention layer and propose the Maximum Singular Value Penalization (MSVP) to precisely manage these bounds By incorporating MSVP into ViTs' attention layers, we enhance the model's robustness without compromising training efficiency. SpecFormer, the resulting model, outperforms other state-of-the-art models in defending against adversarial attacks, as proven by experiments on CIFAR and ImageNet datasets. Code is released at https://github.com/microsoft/robustlearn.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 55 of its 65 references that are in the map. First posted on arXiv 2 Jan 2024.

Builds on

55 of 65 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  10. 2013Intriguing properties of neural networkscited by 5,114
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2014Going Deeper with Convolutionscited by 4,231
38 more
  1. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  2. 2021ViViT: A Video Vision Transformercited by 1,610
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  4. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  5. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  6. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  7. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  8. 2021Going deeper with Image Transformerscited by 759
  9. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  10. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  11. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  12. 2021Intriguing Properties of Vision Transformerscited by 446
  13. 2020Backdoor Learning: A Surveycited by 412
  14. 2017Mitigating Adversarial Effects Through Randomizationcited by 382
  15. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  16. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  17. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  18. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  19. 2017Detecting Adversarial Samples from Artifactscited by 244
  20. 2021Vision Transformers are Robust Learnerscited by 208
  21. 2021Are Transformers More Robust Than CNNs?cited by 187
  22. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 182
  23. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  24. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  25. 2020The Lipschitz Constant of Self-Attentioncited by 171
  26. 2016Improving the Robustness of Deep Neural Networks via Stability Trainingcited by 156
  27. 2017EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examplescited by 154
  28. 2022Understanding The Robustness in Vision Transformerscited by 144
  29. 2021CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptationcited by 129
  30. 2022Robustness and Accuracy Could Be Reconcilable by (Proper) Definitioncited by 109
  31. 2021Globally-Robust Neural Networkscited by 106
  32. 2021On the Adversarial Robustness of Vision Transformerscited by 104
  33. 2016Towards the Science of Security and Privacy in Machine Learningcited by 89
  34. 2022A Light Recipe to Train Robust Vision Transformerscited by 64
  35. 2022When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecturecited by 64
  36. 2018Limitations of the Lipschitz constant as a defense against adversarial examplescited by 48
  37. 2022Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?cited by 47
  38. 2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 38

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  2. 2025Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief Statescited by 9
  3. 2025FOCUS: Fused Observation of Channels for Unveiling Spectracited by 6
  4. 2026Muon in Vision Transformers: Optimizer-Recipe Interactions and Gradient Spectracited by 1
  5. 2025ODE-ViT: Plug & Play Attention Layer from the Generalization of the ViT as an Ordinary Differential Equationcited by 0