mlsanityData to 7 Oct 2026

An Attention Free Transformer

arXiv 2105.14103 ↗First posted 28 May 2021cs.LG cs.CL cs.CVcited by 111

Not on the map. Why?

Abstract (arXiv)

We introduce Attention Free Transformer (AFT), an efficient variant of Transformers that eliminates the need for dot product self attention. In an AFT layer, the key and value are first combined with a set of learned position biases, the result of which is multiplied with the query in an element-wise fashion. This new operation has a memory complexity linear w.r.t. both the context size and the dimension of features, making it compatible to both large input and model sizes. We also introduce AFT-local and AFT-conv, two model variants that take advantage of the idea of locality and spatial weight sharing while maintaining global connectivity. We conduct extensive experiments on two autoregressive modeling tasks (CIFAR10 and Enwik8) as well as an image recognition task (ImageNet-1K classification). We show that AFT demonstrates competitive performance on all the benchmarks, while providing excellent efficiency at the same time.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

30 of 37 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016Layer Normalizationcited by 5,579
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020Reformer: The Efficient Transformercited by 1,658
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
18 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  3. 2020Efficient Transformers: A Surveycited by 784
  4. 2018Image Transformercited by 770
  5. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  6. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  7. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  8. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  9. 2021Pay Attention to MLPscited by 367
  10. 2021Random Feature Attentioncited by 350
  11. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  12. 2020Sparse Sinkhorn Attentioncited by 250
  13. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  14. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  15. 2019Asymmetric Non-local Neural Networks for Semantic Segmentationcited by 196
  16. 2019Adaptive Attention Span in Transformerscited by 148
  17. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  18. 2019Interlaced Sparse Self-Attention for Semantic Segmentationcited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 15; 2023: 22; 2024: 45; 2025: 15; 2026: 11.

111 papers.

Sort:

By area:

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  4. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  5. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  6. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  8. 2021ResViT: Residual vision transformers for multi-modal medical image synthesiscited by 177
  9. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  10. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  11. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 142
  12. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
38 more
  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  2. 2022Filter-enhanced MLP is All You Need for Sequential Recommendationcited by 96
  3. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  4. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  5. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  6. 2023SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networkscited by 82
  7. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  8. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  9. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 65
  10. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  11. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  12. 2025End-to-End Test-Time Training for Long Contextcited by 53
  13. 2024Linearizing Large Language Modelscited by 51
  14. 2024A Survey on Visual Mambacited by 50
  15. 2024A Survey on Transformer Compressioncited by 44
  16. 2022Temporally Consistent Transformers for Video Generationcited by 43
  17. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  18. 2023Diffusion Probabilistic Fieldscited by 28
  19. 2022Fine-Tuning Pre-trained Transformers into Decaying Fast Weightscited by 26
  20. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  21. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  22. 2024DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Modelscited by 22
  23. 2024Scalable MatMul-free Language Modelingcited by 21
  24. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  25. 2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 20
  26. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 20
  27. 2024On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Eracited by 17
  28. 2023Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Abilitycited by 16
  29. 2024DiJiang: Efficient Large Language Models through Compact Kernelizationcited by 15
  30. 2023Blockwise Parallel Transformer for Large Context Modelscited by 12
  31. 2024Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysiscited by 12
  32. 2023Manifold Diffusion Fieldscited by 11
  33. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  34. 2022FeatureBooster: Boosting Feature Descriptors with a Lightweight Neural Networkcited by 10
  35. 2022FsaNet: Frequency Self-attention for Semantic Segmentationcited by 10
  36. 2023AutoFocusFormer: Image Segmentation off the Gridcited by 9
  37. 2024Instance-Conditioned Adaptation for Large-scale Generalization of Neural Routing Solvercited by 9
  38. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9

50 most cited of 111 All 111 (JSON)