# Exploring Corruption Robustness: Inductive Biases in Vision Transformers and MLP-Mixers

arXiv: https://arxiv.org/abs/2106.13122 · HTML: https://mlsanity.com/paper/2106.13122/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 21 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)
- [Shape-Texture Debiased Neural Network Training](https://mlsanity.com/paper/2010.05981/) (2020)
- [Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study](https://mlsanity.com/paper/1706.08606/) (2017)
- [Does enhanced shape bias improve neural network robustness to common corruptions?](https://mlsanity.com/paper/2104.09789/) (2021)
- [Learning Inductive Biases with Simple Neural Networks](https://mlsanity.com/paper/1802.02745/) (2018)
- [The Origins and Prevalence of Texture Bias in Convolutional Neural Networks](https://mlsanity.com/paper/1911.09071/) (2019)
- [Increasing Shape Bias in ImageNet-Trained Networks Using Transfer Learning and Domain-Adversarial Methods](https://mlsanity.com/paper/1907.12892/) (2019)

## Built on by (most cited first, 13 of 13)

- [Are we ready for a new paradigm shift? A Survey on Visual Deep MLP](https://mlsanity.com/paper/2111.04060/) (2021)
- [PRIME: A few primitives can boost robustness to common corruptions](https://mlsanity.com/paper/2112.13547/) (2021)
- [An Impartial Take to the CNN vs Transformer Robustness Contest](https://mlsanity.com/paper/2207.11347/) (2022)
- [Robustifying Vision Transformer without Retraining from Scratch by Test-Time Class-Conditional Feature Alignment](https://mlsanity.com/paper/2206.13951/) (2022)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [Deep Digging into the Generalization of Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2205.11083/) (2022)
- [Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones](https://mlsanity.com/paper/2205.01586/) (2022)
- [Improving Object Detector Training on Synthetic Data by Starting With a Strong Baseline Methodology](https://mlsanity.com/paper/2405.19822/) (2024)
- [A Study on the Generality of Neural Network Structures for Monocular Depth Estimation](https://mlsanity.com/paper/2301.03169/) (2023)
- [Harmonized Spatial and Spectral Learning for Robust and Generalized Medical Image Segmentation](https://mlsanity.com/paper/2401.10373/) (2024)
- [Colonoscopy Landmark Detection using Vision Transformers](https://mlsanity.com/paper/2209.11304/) (2022)
- [Dissecting Continual Learning a Structural and Data Analysis](https://mlsanity.com/paper/2301.01033/) (2023)
- [MAPS: A Synthetic Dataset for Probing Vision Models in a Controlled 3D Scene Space](https://mlsanity.com/paper/2605.20549/) (2026)

## Abstract (arXiv)

Recently, vision transformers and MLP-based models have been developed in order to address some of the prevalent weaknesses in convolutional neural networks. Due to the novelty of transformers being used in this domain along with the self-attention mechanism, it remains unclear to what degree these architectures are robust to corruptions. Despite some works proposing that data augmentation remains essential for a model to be robust against corruptions, we propose to explore the impact that the architecture has on corruption robustness. We find that vision transformer architectures are inherently more robust to corruptions than the ResNet-50 and MLP-Mixers. We also find that vision transformers with 5 times fewer parameters than a ResNet-50 have more shape bias. Our code is available to reproduce.