mlsanityData to 7 Oct 2026

Replacing softmax with ReLU in Vision Transformers

arXiv 2309.08586 ↗First posted 15 Sep 2023cs.CV cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU. In the context of vision transformers, we find that this degradation is mitigated when dividing by sequence length. Our experiments training small to large vision transformers on ImageNet-21k indicate that ReLU-attention can approach or match the performance of softmax-attention in terms of scaling behavior as a function of compute.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

19 of 30 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Layer Normalizationcited by 5,579
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  10. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  11. 2022Transformer Quality in Linear Timecited by 189
  12. 2021Self-attention Does Not Need O(n2) Memorycited by 152
7 more
  1. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  2. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  3. 2022Better plain ViT baselines for ImageNet-1kcited by 97
  4. 2023A Study on ReLU and Softmax in Transformercited by 53
  5. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  6. 2022SimA: Simple Softmax-free Attention for Vision Transformerscited by 32
  7. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 13; 2025: 8; 2026: 14.

43 papers.

Sort:

By area:

  1. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  2. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  3. 2024The Hidden Attention of Mamba Modelscited by 88
  4. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  5. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  6. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  7. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  8. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  9. 2023MobileDiffusion: Instant Text-to-Image Generation on Mobile Devicescited by 42
  10. 2024Frequency Decoupling for Motion Magnification via Multi-Level Isomorphic Architecturecited by 28
  11. 2024On Provable Length and Compositional Generalizationcited by 22
  12. 2024HSR-Enhanced Sparse Attention Accelerationcited by 19
31 more
  1. 2024Attention layers provably solve single-location regressioncited by 15
  2. 2024More Expressive Attention with Negative Weightscited by 9
  3. 2024Rethinking Attention: Polynomial Alternatives to Softmax in Transformerscited by 8
  4. 2023MobileUtr: Revisiting the relationship between light-weight CNN and Transformer for efficient medical image segmentationcited by 4
  5. 2024Attention is a smoothed cubic splinecited by 4
  6. 2026ZeroS: Zero-Sum Linear Attention for Efficient Transformerscited by 4
  7. 2024NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMscited by 3
  8. 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approachcited by 3
  9. 2025Self-Adjust Softmaxcited by 3
  10. 2025Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformerscited by 3
  11. 2024Optimized Speculative Sampling for GPU Hardware Acceleratorscited by 2
  12. 2025Efficient High-Resolution Image Editing with Hallucination-Aware Loss and Adaptive Tilingcited by 2
  13. 2026HyperMLP: An Integrated Perspective for Sequence Modelingcited by 2
  14. 2024USEFUSE: Uniform Stride for Enhanced Performance in Fused Layer Architecture of Deep Neural Networkscited by 1
  15. 2025What Can We Learn From MIMO Graph Convolutions?cited by 1
  16. 2025Rethinking Graph-Based Document Classification: Learning Data-Driven Structures Beyond Heuristic Approachescited by 1
  17. 2026Capacity-Controlled Global Attention for Graph Transformerscited by 1
  18. 2026Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attentioncited by 1
  19. 2024Sampling Foundational Transformer: A Theoretical Perspectivecited by 0
  20. 2025Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Modelscited by 0
  21. 2025Lightweight Backbone Networks Only Require Adaptive Lightweight Self-Attention Mechanismscited by 0
  22. 2026Finite-Time Analysis of Gradient Descent for Shallow Transformerscited by 0
  23. 2026ReGLA: Efficient Receptive-Field Modeling with Gated Linear Attention Networkcited by 0
  24. 2026From Global to Local: Learning Context-Aware Graph Representations for Document Classification and Summarizationcited by 0
  25. 2026Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutionscited by 0
  26. 2026A Boltzmann-machine-enhanced Transformer For DNA Sequence Classificationcited by 0
  27. 2026Cubit: Token Mixer with Kernel Ridge Regressioncited by 0
  28. 2026A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventionscited by 0
  29. 2026Pretraining Transformers with Quantized Softmax in Attentioncited by 0
  30. 2026Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contextscited by 0
  31. 2026Attention Kernels for Learning Maps Between Heavy-Tailed Measurescited by 0