mlsanityData to 7 Oct 2026

Transformers Learn Low Sensitivity Functions: Investigations and Implications

arXiv 2403.06925 ↗First posted 11 Mar 2024cs.LG cs.AI cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Transformers achieve state-of-the-art accuracy and robustness across many tasks, but an understanding of their inductive biases and how those biases differ from other neural network architectures remains elusive. In this work, we identify the sensitivity of the model to token-wise random perturbations in the input as a unified metric which explains the inductive bias of transformers across different data modalities and distinguishes them from other architectures. We show that transformers have lower sensitivity than MLPs, CNNs, ConvMixers and LSTMs, across both vision and language tasks. We also show that this low-sensitivity bias has important implications: i) lower sensitivity correlates with improved robustness; it can also be used as an efficient intervention to further improve the robustness of transformers; ii) it corresponds to flatter minima in the loss landscape; and iii) it can serve as a progress measure for grokking. We support these findings with theoretical results showing (weak) spectral bias of transformers in the NTK regime, and improved robustness due to the lower sensitivity. The code is available at https://github.com/estija/sensitivity.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 94 of its 118 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

94 of 118 references.

References · 94

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  11. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  12. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
38 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2018On the Spectral Bias of Neural Networkscited by 1,321
  3. 2017A Closer Look at Memorization in Deep Networkscited by 936
  4. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  5. 2022Transformers learn in-context by gradient descentcited by 714
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  7. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  8. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  10. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  11. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  12. 2017Deep Neural Networks as Gaussian Processescited by 617
  13. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  14. 2017Exploring Generalization in Deep Learningcited by 595
  15. 2021Intriguing Properties of Vision Transformerscited by 446
  16. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  17. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  18. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  19. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  20. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  21. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  22. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  23. 2019Revealing the Dark Secrets of BERTcited by 300
  24. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  25. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  26. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  27. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  28. 2019Fast AutoAugmentcited by 243
  29. 2021Vision Transformers are Robust Learnerscited by 208
  30. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  31. 2022Patches Are All You Need?cited by 201
  32. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  33. 2021Data Augmentation Can Improve Robustnesscited by 193
  34. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  35. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  36. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  37. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  38. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 6.

13 papers.

Sort:

By area:

  1. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 37
  2. 2025Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformerscited by 28
  3. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 16
  4. 2024Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selectioncited by 9
  5. 2024Spectral Convolutional Conditional Neural Processescited by 4
  6. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  7. 2026From Dispersion to Attraction: Spectral Dynamics of Hallucination Across Whisper Model Scalescited by 2
  8. 2025Transformers with RL or SFT Provably Learn Sparse Boolean Functions, But Differentlycited by 1
  9. 2026Trapped by simplicity: When Transformers fail to learn from noisy featurescited by 0
  10. 2026Not All NVFP4 QAT Recipes Are Equal: How Architecture and Scale Shape Model Quality for Anomaly Segmentationcited by 0
  11. 2026Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Biascited by 0
  12. 2026Revisiting Neural Processes via Fourier Transform and Volterra Seriescited by 0
1 more
  1. 2026Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatchcited by 0