mlsanityData to 14 Oct 2026

Are Vision Transformers Robust to Spurious Correlations?

arXiv 2203.09125 ↗First posted 17 Mar 2022cs.CV cs.AI cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Deep neural networks may be susceptible to learning spurious correlations that hold on average but not in atypical test samples. As with the recent emergence of vision transformer (ViT) models, it remains underexplored how spurious correlations are manifested in such architectures. In this paper, we systematically investigate the robustness of vision transformers to spurious correlations on three challenging benchmark datasets and compare their performance with popular CNNs. Our study reveals that when pre-trained on a sufficiently large dataset, ViT models are more robust to spurious correlations than CNNs. Key to their success is the ability to generalize better from the examples where spurious correlations do not hold. Further, we perform extensive ablations and experiments to understand the role of the self-attention mechanism in providing robustness under spuriously correlated environments. We hope that our work will inspire future research on further understanding the robustness of ViT models.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 49 of its 64 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

49 of 64 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  12. 2022A ConvNet for the 2020scited by 3,579
37 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  9. 2019Natural Adversarial Examplescited by 1,408
  10. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  11. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  13. 2021Scaling Vision Transformerscited by 999
  14. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  15. 2020Quantifying Attention Flow in Transformerscited by 775
  16. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  17. 2021Going deeper with Image Transformerscited by 764
  18. 2018Recognition in Terra Incognitacited by 658
  19. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  20. 2021Intriguing Properties of Vision Transformerscited by 446
  21. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  22. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  23. 2022How Do Vision Transformers Work?cited by 307
  24. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  25. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  26. 2021Vision Transformers are Robust Learnerscited by 207
  27. 2021Are Transformers More Robust Than CNNs?cited by 187
  28. 2016Grad-CAM: Why did you say that?cited by 161
  29. 2021Towards Robust Vision Transformercited by 137
  30. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 134
  31. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 123
  32. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  33. 2021Delving Deep into the Generalization of Vision Transformers under Distribution Shiftscited by 85
  34. 2020Model Patching: Closing the Subgroup Performance Gap with Data Augmentationcited by 83
  35. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 75
  36. 2021On the Impact of Spurious Correlation for Out-of-distribution Detectioncited by 68
  37. 2021Go Wider Instead of Deepercited by 57

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 6; 2024: 14; 2025: 10; 2026: 2.

36 papers.

Sort:

By area:

  1. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  2. 2024ARGS: Alignment as Reward-Guided Searchcited by 103
  3. 2024Process Reward Model with Q-Value Rankingscited by 80
  4. 2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
  5. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMscited by 33
  6. 2022ModelDiff: A Framework for Comparing Learning Algorithmscited by 30
  7. 2022How Robust is Unsupervised Representation Learning to Distribution Shift?cited by 24
  8. 2022How to Fine-Tune Vision Models with SGDcited by 23
  9. 2023Distributionally Robust Post-hoc Classifiers under Prior Shiftscited by 16
  10. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 14
  11. 2023FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learningcited by 12
  12. 2024Controllable Prompt Tuning For Balancing Group Distributional Robustnesscited by 10
24 more
  1. 2024Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapescited by 10
  2. 2024On the Learnability of Out-of-distribution Detectioncited by 8
  3. 2023Fairness-aware Vision Transformer via Debiased Self-Attentioncited by 6
  4. 2024Benchmarking Spurious Bias in Few-Shot Image Classifierscited by 6
  5. 2024Mask of truth: model sensitivity to unexpected regions of medical imagescited by 6
  6. 2025Efficient Unsupervised Shortcut Learning Detection and Mitigation in Transformerscited by 6
  7. 2024Learning Generalizable Models via Disentangling Spurious and Enhancing Potential Correlationscited by 5
  8. 2024SLIM: Spuriousness Mitigation with Minimal Human Annotationscited by 5
  9. 2025Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applicationscited by 4
  10. 2023Identifying and Disentangling Spurious Features in Pretrained Image Representationscited by 3
  11. 2025The Devil is in the Spurious Correlations: Boosting Moment Retrieval with Dynamic Learningcited by 3
  12. 2025SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrievalcited by 2
  13. 2025Moment Alignment: Unifying Gradient and Hessian Matching for Domain Generalizationcited by 2
  14. 2023Depth-Relative Self Attention for Monocular Depth Estimationcited by 1
  15. 2025Single-weight Model Editing for Post-hoc Spurious Correlation Neutralizationcited by 1
  16. 2025Spurious Privacy Leakage in Neural Networkscited by 1
  17. 2026SCL-GNN: Towards Generalizable Graph Neural Networks via Spurious Correlation Learningcited by 1
  18. 2024Long-Term Prediction Accuracy Improvement of Data-Driven Medium-Range Global Weather Forecastcited by 0
  19. 2024A Survey on Cell Nuclei Instance Segmentation and Classification: Leveraging Context and Attentioncited by 0
  20. 2024The Pragmatic Frames of Spurious Correlations in Machine Learning: Interpreting How and Why They Mattercited by 0
  21. 2025Out-of-distribution data supervision towards biomedical semantic segmentationcited by 0
  22. 2025Token-Based Detection of Spurious Correlations in Vision Transformerscited by 0
  23. 2025IGAff: Benchmarking Adversarial Iterative and Genetic Affine Algorithms on Deep Neural Networkscited by 0
  24. 2026Deep Attention Reweighting: Post-Hoc Attention-Based Feature Aggregation in CNNs for Disentangling Core and Spurious Features under Spurious Correlationscited by 0