mlsanityData to 7 Oct 2026

Exploring Corruption Robustness: Inductive Biases in Vision Transformers and MLP-Mixers

arXiv 2106.13122 ↗First posted 24 Jun 2021cs.CV cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Recently, vision transformers and MLP-based models have been developed in order to address some of the prevalent weaknesses in convolutional neural networks. Due to the novelty of transformers being used in this domain along with the self-attention mechanism, it remains unclear to what degree these architectures are robust to corruptions. Despite some works proposing that data augmentation remains essential for a model to be robust against corruptions, we propose to explore the impact that the architecture has on corruption robustness. We find that vision transformer architectures are inherently more robust to corruptions than the ResNet-50 and MLP-Mixers. We also find that vision transformers with 5 times fewer parameters than a ResNet-50 have more shape bias. Our code is available to reproduce.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 19 of its 21 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

19 of 21 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  11. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  12. 2021Going deeper with Image Transformerscited by 759
7 more
  1. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  2. 2020Shape-Texture Debiased Neural Network Trainingcited by 76
  3. 2017Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Studycited by 46
  4. 2021Does enhanced shape bias improve neural network robustness to common corruptions?cited by 32
  5. 2018Learning Inductive Biases with Simple Neural Networkscited by 8
  6. 2019The Origins and Prevalence of Texture Bias in Convolutional Neural Networkscited by 6
  7. 2019Increasing Shape Bias in ImageNet-Trained Networks Using Transfer Learning and Domain-Adversarial Methodscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 3; 2024: 2; 2025: 0; 2026: 1.

13 papers.

Sort:

  1. 2021Are we ready for a new paradigm shift? A Survey on Visual Deep MLPcited by 63
  2. 2021PRIME: A few primitives can boost robustness to common corruptionscited by 40
  3. 2022An Impartial Take to the CNN vs Transformer Robustness Contestcited by 34
  4. 2022Robustifying Vision Transformer without Retraining from Scratch by Test-Time Class-Conditional Feature Alignmentcited by 26
  5. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  6. 2022Deep Digging into the Generalization of Self-Supervised Monocular Depth Estimationcited by 24
  7. 2022Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbonescited by 12
  8. 2024Improving Object Detector Training on Synthetic Data by Starting With a Strong Baseline Methodologycited by 6
  9. 2023A Study on the Generality of Neural Network Structures for Monocular Depth Estimationcited by 4
  10. 2024Harmonized Spatial and Spectral Learning for Robust and Generalized Medical Image Segmentationcited by 2
  11. 2022Colonoscopy Landmark Detection using Vision Transformerscited by 1
  12. 2023Dissecting Continual Learning a Structural and Data Analysiscited by 0
1 more
  1. 2026MAPS: A Synthetic Dataset for Probing Vision Models in a Controlled 3D Scene Spacecited by 0