mlsanityData to 7 Oct 2026

Understanding The Robustness in Vision Transformers

arXiv 2204.12451 ↗First posted 26 Apr 2022cs.CVcited by 144

Not on the map. Why?

Abstract (arXiv)

Recent studies show that Vision Transformers(ViTs) exhibit strong robustness against various corruptions. Although this property is partly attributed to the self-attention mechanism, there is still a lack of systematic understanding. In this paper, we examine the role of self-attention in learning robust representations. Our study is motivated by the intriguing properties of the emerging visual grouping in Vision Transformers, which indicates that self-attention may promote robustness through improved mid-level representations. We further propose a family of fully attentional networks (FANs) that strengthen this capability by incorporating an attentional channel processing design. We validate the design comprehensively on various hierarchical backbones. Our model achieves a state-of-the-art 87.1% accuracy and 35.8% mCE on ImageNet-1k and ImageNet-C with 76.8M parameters. We also demonstrate state-of-the-art accuracy and robustness in two downstream tasks: semantic segmentation and object detection. Code is available at: https://github.com/NVlabs/FAN.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). Builds on 46 of its 61 references that are in the map. First posted on arXiv 26 Apr 2022.

Builds on

46 of 61 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2017Mask R-CNNcited by 7,259
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  12. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
34 more
  1. 2017Squeeze-and-Excitation Networkscited by 4,763
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2022A ConvNet for the 2020scited by 3,545
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  6. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2016Pyramid Scene Parsing Networkcited by 2,472
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  10. 2019Natural Adversarial Examplescited by 1,401
  11. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  12. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  13. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  14. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  15. 2016Adversarial Machine Learning at Scalecited by 934
  16. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  17. 2021Going deeper with Image Transformerscited by 759
  18. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  19. 2021Intriguing Properties of Vision Transformerscited by 446
  20. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  21. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  22. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  23. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  24. 2017ICNet for Real-Time Semantic Segmentation on High-Resolution Imagescited by 261
  25. 2021Vision Transformers are Robust Learnerscited by 208
  26. 2021Are Transformers More Robust Than CNNs?cited by 187
  27. 2016Wider or Deeper: Revisiting the ResNet Model for Visual Recognitioncited by 184
  28. 2021Towards Robust Vision Transformercited by 137
  29. 2019Benchmarking the Robustness of Semantic Segmentation Modelscited by 111
  30. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  31. 2021On the Adversarial Robustness of Vision Transformerscited by 104
  32. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  33. 2021Refiner: Refining Self-attention for Vision Transformerscited by 38
  34. 2021Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?cited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 26; 2023: 43; 2024: 35; 2025: 23; 2026: 17.

144 papers.

Sort:

By area:

  1. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  2. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  3. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  4. 2022MetaFormer Baselines for Visioncited by 131
  5. 2023Dual Aggregation Transformer for Image Super-Resolutioncited by 124
  6. 2022Diffusion Probabilistic Model Made Slimcited by 121
  7. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 115
  8. 2022Sharpness-Aware Training for Freecited by 95
  9. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  10. 2022Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarioscited by 65
  11. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  12. 2023How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defensescited by 65
38 more
  1. 2023TransNeXt: Robust Foveal Visual Perception for Vision Transformerscited by 61
  2. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  3. 2022Towards Robust Referring Image Segmentationcited by 49
  4. 2023Scale-Aware Modulation Meet Transformercited by 49
  5. 2023Dataset Quantizationcited by 47
  6. 2022Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentationcited by 46
  7. 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
  8. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  9. 2024DRCT: Saving Image Super-resolution away from Information Bottleneckcited by 40
  10. 2023Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Wordscited by 33
  11. 2022Imaging through the Atmosphere using Turbulence Mitigation Transformercited by 31
  12. 2023HGFormer: Hierarchical Grouping Transformer for Domain Generalized Semantic Segmentationcited by 31
  13. 2024AutoGluon-Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Modelscited by 29
  14. 2023Lightweight Vision Transformer with Bidirectional Interactioncited by 25
  15. 2023Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNNcited by 25
  16. 2022Exploring the Relationship between Architecture and Adversarially Robust Generalizationcited by 23
  17. 2024Understanding Video Transformers via Universal Concept Discoverycited by 22
  18. 2023A Survey on the Robustness of Computer Vision Models against Common Corruptionscited by 20
  19. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  20. 2022WaveMix: A Resource-efficient Neural Network for Image Analysiscited by 19
  21. 2023Robustifying Token Attention for Vision Transformerscited by 19
  22. 2023Top-Down Visual Attention from Analysis by Synthesiscited by 19
  23. 2022Discrete Key-Value Bottleneckcited by 18
  24. 2023Adaptive Sharpness-Aware Pruning for Robust Sparse Networkscited by 17
  25. 2023Reliability in Semantic Segmentation: Are We on the Right Track?cited by 16
  26. 2023Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attentioncited by 15
  27. 2024Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metricscited by 15
  28. 2023Vision Transformers Are Good Mask Auto-Labelerscited by 14
  29. 2024Investigating Out-of-Distribution Generalization of GNNs: An Architecture Perspectivecited by 13
  30. 2022Comparing the Decision-Making Mechanisms by Transformers and CNNs via Explanation Methodscited by 12
  31. 2023Low-Resolution Self-Attention for Semantic Segmentationcited by 12
  32. 2023Improving Interpretation Faithfulness for Vision Transformerscited by 12
  33. 2024RoDLA: Benchmarking the Robustness of Document Layout Analysis Modelscited by 12
  34. 2024Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptionscited by 12
  35. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Modelscited by 12
  36. 2022Designing Robust Transformers using Robust Kernel Density Estimationcited by 11
  37. 2023Improving Adversarial Transferability via Model Alignmentcited by 10
  38. 2024PIDformer: Transformer Meets Control Theorycited by 10

50 most cited of 144 All 144 (JSON)