mlsanityData to 14 Oct 2026

Your Attention Matters: to Improve Model Robustness to Noise and Spurious Correlations

arXiv 2507.20453 ↗First posted 28 Jul 2025cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Self-attention mechanisms are foundational to Transformer architectures, supporting their impressive success in a wide range of tasks. While there are many self-attention variants, their robustness to noise and spurious correlations has not been well studied. This study evaluates Softmax, Sigmoid, Linear, Doubly Stochastic, and Cosine attention within Vision Transformers under different data corruption scenarios. Through testing across the CIFAR-10, CIFAR-100, and Imagenette datasets, we show that Doubly Stochastic attention is the most robust. It consistently outperformed the next best mechanism by 0.1%-5.1% when training data, or both training and testing data, were corrupted. Our findings inform self-attention selection in contexts with imperfect data. The code used is available at https://github.com/ctamayor/NeurIPS-Robustness-ViT.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

31 of 40 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
19 more
  1. 2014Going Deeper with Convolutionscited by 4,241
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  6. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  7. 2020Rethinking Attention with Performerscited by 1,601
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  9. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  11. 2022A Generalist Agentcited by 808
  12. 2021Going deeper with Image Transformerscited by 764
  13. 2021AST: Audio Spectrogram Transformercited by 643
  14. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  15. 2021Vision Transformers are Robust Learnerscited by 207
  16. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  17. 2024Theory, Analysis, and Best Practices for Sigmoid Self-Attentioncited by 64
  18. 2020Doubly-Stochastic Normalization of the Gaussian Kernel is Robust to Heteroskedastic Noisecited by 16
  19. 2024Cottention: Linear Transformers With Cosine Attentioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Taskscited by 2
  2. 2026CLIMP: Contrastive Language-Image Mamba Pretrainingcited by 1
  3. 2026PhysAttNet: Enhancing Predictive Performance in Industrial and Astrophysical Time Series via Physics-Informed Attentioncited by 0