mlsanityData to 7 Oct 2026

Towards Robust Vision Transformer

arXiv 2105.07926 ↗First posted 17 May 2021cs.CVcited by 137

Not on the map. Why?

Abstract (arXiv)

Recent advances on Vision Transformer (ViT) and its improved variants have shown that self-attention-based networks surpass traditional Convolutional Neural Networks (CNNs) in most vision tasks. However, existing ViTs focus on the standard accuracy and computation cost, lacking the investigation of the intrinsic influence on model robustness and generalization. In this work, we conduct systematic evaluation on components of ViTs in terms of their impact on robustness to adversarial examples, common corruptions and distribution shifts. We find some components can be harmful to robustness. By using and combining robust components as building blocks of ViTs, we propose Robust Vision Transformer (RVT), which is a new vision transformer and has superior performance with strong robustness. We further propose two new plug-and-play techniques called position-aware attention scaling and patch-wise augmentation to augment our RVT, which we abbreviate as RVT*. The experimental results on ImageNet and six robustness benchmarks show the advanced robustness and generalization ability of RVT compared with previous ViTs and state-of-the-art CNNs. Furthermore, RVT-S* also achieves Top-1 rank on multiple robustness leaderboards including ImageNet-C and ImageNet-Sketch. The code will be available at urlhttps://github.com/alibaba/easyrobust.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Builds on 46 of its 52 references that are in the map. First posted on arXiv 17 May 2021.

Builds on

46 of 52 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2013Intriguing properties of neural networkscited by 5,114
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
34 more
  1. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  2. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2019Natural Adversarial Examplescited by 1,401
  6. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  10. 2020Designing Network Design Spacescited by 954
  11. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  12. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  13. 2019Are Sixteen Heads Really Better than One?cited by 803
  14. 2021Transformer in Transformercited by 632
  15. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  16. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  17. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  18. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  19. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  20. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  21. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 231
  22. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  23. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  24. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  25. 2018Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Modelscited by 135
  26. 2019Adversarial Robustness vs Model Compression, or Both?cited by 98
  27. 2019Defensive Quantization: When Efficiency Meets Robustnesscited by 95
  28. 2020Smooth Adversarial Trainingcited by 89
  29. 2019When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attackscited by 87
  30. 2020Do Wider Neural Networks Really Help Adversarial Robustness?cited by 79
  31. 2020Shape-Texture Debiased Neural Network Trainingcited by 76
  32. 2016Inductive Bias of Deep Convolutional Networks through Pooling Geometrycited by 53
  33. 2020An Effective Anti-Aliasing Approach for Residual Networkscited by 23
  34. 2020Adversarially Robust Neural Architecturescited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 36; 2023: 33; 2024: 30; 2025: 21; 2026: 7.

137 papers.

Sort:

By area:

  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2022A ConvNet for the 2020scited by 3,545
  3. 2022Multimodal Learning with Transformers: A Surveycited by 399
  4. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  5. 2022How Do Vision Transformers Work?cited by 305
  6. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  7. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  8. 2022Understanding The Robustness in Vision Transformerscited by 144
  9. 2022Towards Adversarial Attack on Vision-Language Pre-training Modelscited by 142
  10. 2022MetaFormer Baselines for Visioncited by 131
  11. 2023RepViT: Revisiting Mobile CNN From ViT Perspectivecited by 123
  12. 2022MobileOne: An Improved One millisecond Mobile Backbonecited by 109
38 more
  1. 2023MedViT: A Robust Vision Transformer for Generalized Medical Image Classificationcited by 89
  2. 2021TransMix: Attend to Mix for Vision Transformerscited by 79
  3. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  4. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  5. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  6. 2023How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defensescited by 65
  7. 2022A Light Recipe to Train Robust Vision Transformerscited by 64
  8. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  9. 2022Pro-tuning: Unified Prompt Tuning for Vision Taskscited by 53
  10. 2022MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Featurescited by 44
  11. 2021Are Vision Transformers Robust to Patch Perturbations?cited by 39
  12. 2021Pyramid Adversarial Training Improves ViT Performancecited by 39
  13. 2025UniRestore: Unified Perceptual and Task-Oriented Image Restoration Model Using Diffusion Priorcited by 39
  14. 2022Are Vision Transformers Robust to Spurious Correlations?cited by 36
  15. 2022DETRDistill: A Universal Knowledge Distillation Framework for DETR-familiescited by 36
  16. 2021Discrete Representations Strengthen Vision Transformer Robustnesscited by 34
  17. 2023COCO-O: A Benchmark for Object Detectors under Natural Distribution Shiftscited by 34
  18. 2023Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Wordscited by 33
  19. 2022Towards Efficient Adversarial Training on Vision Transformerscited by 29
  20. 2023Survey on Unsupervised Domain Adaptation for Semantic Segmentation for Visual Perception in Automated Drivingcited by 29
  21. 2024Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learningcited by 29
  22. 2022Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasetscited by 28
  23. 2022Context-Aware Robust Fine-Tuningcited by 27
  24. 2023Masked Contrastive Graph Representation Learning for Age Estimationcited by 27
  25. 2022Towards Compositional Adversarial Robustness: Generalizing Adversarial Training to Composite Semantic Perturbationscited by 26
  26. 2023PAIF: Perception-Aware Infrared-Visible Image Fusion for Attack-Tolerant Semantic Segmentationcited by 26
  27. 2022Sequencer: Deep LSTM for Image Classificationcited by 23
  28. 2022Jigsaw-ViT: Learning Jigsaw Puzzles in Vision Transformercited by 21
  29. 2023A Survey on the Robustness of Computer Vision Models against Common Corruptionscited by 20
  30. 2021Sparse MoEs meet Efficient Ensemblescited by 19
  31. 2022Defending Backdoor Attacks on Vision Transformer via Patch Processingcited by 19
  32. 2023Robustifying Token Attention for Vision Transformerscited by 19
  33. 2023Top-Down Visual Attention from Analysis by Synthesiscited by 19
  34. 2021Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removalcited by 16
  35. 2023Reliability in Semantic Segmentation: Are We on the Right Track?cited by 16
  36. 2024Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Modelscited by 16
  37. 2025LSNet: See Large, Focus Smallcited by 15
  38. 2023Revisiting and Exploring Efficient Fast Adversarial Training via LAW: Lipschitz Regularization and Auto Weight Averagingcited by 14

50 most cited of 137 All 137 (JSON)