mlsanityData to 7 Oct 2026

Fastformer: Additive Attention Can Be All You Need

arXiv 2108.09084 ↗First posted 20 Aug 2021cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

Transformer is a powerful model for text understanding. However, it is inefficient due to its quadratic complexity to input sequence length. Although there are many methods on Transformer acceleration, they are still either inefficient on long sequences or not effective enough. In this paper, we propose Fastformer, which is an efficient Transformer model based on additive attention. In Fastformer, instead of modeling the pair-wise interactions between tokens, we first use additive attention mechanism to model global contexts, and then further transform each token representation based on its interaction with global context representations. In this way, Fastformer can achieve effective context modeling with linear complexity. Extensive experiments on five datasets show that Fastformer is much more efficient than many existing Transformer models and can meanwhile achieve comparable or even better long text modeling performance.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 20 Aug 2021.

Builds on

23 of 35 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2015Teaching Machines to Read and Comprehendcited by 1,039
11 more
  1. 2020Efficient Transformers: A Surveycited by 784
  2. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  3. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  4. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  5. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  6. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  7. 2021Empowering News Recommendation with Pre-trained Language Modelscited by 63
  8. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  9. 2020Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformerscited by 49
  10. 2021Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modelingcited by 35
  11. 2017Item Silk Road: Recommending Items from Information Domains to Social Userscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 18; 2023: 18; 2024: 9; 2025: 8; 2026: 5.

61 papers.

Sort:

By area:

  1. 2022Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentationcited by 114
  2. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  3. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  4. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  5. 2023SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applicationscited by 66
  6. 2023ONCE: Boosting Content-based Recommendation with Both Open- and Closed-source Large Language Modelscited by 54
  7. 2022CATs++: Boosting Cost Aggregation with Convolutions and Transformerscited by 53
  8. 2022TransforMatcher: Match-to-Match Attention for Semantic Correspondencecited by 29
  9. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  10. 2022Boosting Robustness of Image Matting with Context Assembling and Strong Data Augmentationcited by 22
  11. 2023DeepMatcher: A Deep Transformer-based Network for Robust and Accurate Local Feature Matchingcited by 18
  12. 2021Cost Aggregation Is All You Need for Few-Shot Segmentationcited by 16
38 more
  1. 2024Unifying Feature and Cost Aggregation with Transformers for Semantic and Visual Correspondencecited by 14
  2. 2022DCT-Former: Efficient Self-Attention with Discrete Cosine Transformcited by 12
  3. 2023SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understandingcited by 11
  4. 2023HOT: Higher-Order Dynamic Graph Representation Learning with Efficient Transformerscited by 10
  5. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  6. 2025On-device edge learning for IoT data streams: a surveycited by 8
  7. 2022Image Search with Text Feedback by Additive Attention Compositional Learningcited by 7
  8. 2023OAMatcher: An Overlapping Areas-based Network for Accurate Local Feature Matchingcited by 7
  9. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 7
  10. 2022Integrative Feature and Cost Aggregation with Transformers for Dense Correspondencecited by 6
  11. 2021PoNet: Pooling Network for Efficient Token Mixing in Long Sequencescited by 5
  12. 2022Transformer-based Value Function Decomposition for Cooperative Multi-agent Reinforcement Learning in StarCraftcited by 5
  13. 2022Uconv-Conformer: High Reduction of Input Sequence Length for End-to-End Speech Recognitioncited by 5
  14. 2021Audiomer: A Convolutional Transformer For Keyword Spottingcited by 4
  15. 2023AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarioscited by 4
  16. 2023RADIFUSION: A multi-radiomics deep learning based breast cancer risk prediction model using sequential mammographic images with image attention and bilateral asymmetry refinementcited by 4
  17. 2022GRAM: Fast Fine-tuning of Pre-trained Language Models for Content-based Collaborative Filteringcited by 3
  18. 2022VLSNR:Vision-Linguistics Coordination Time Sequence-aware News Recommendationcited by 3
  19. 2023ConvFormer: Revisiting Transformer for Sequential User Modelingcited by 3
  20. 2023Rethinking Urban Mobility Prediction: A Super-Multivariate Time Series Forecasting Approachcited by 3
  21. 2022A Novel Perspective to Look At Attention: Bi-level Attention-based Explainable Topic Modeling for News Classificationcited by 2
  22. 2023Efficient Joint Learning for Clinical Named Entity Recognition and Relation Extraction Using Fourier Networks: A Use Case in Adverse Drug Eventscited by 2
  23. 2023Inhibitor Transformers and Gated RNNs for Torus Efficient Fully Homomorphic Encryptioncited by 2
  24. 2023FMRT: Learning Accurate Feature Matching with Reconciliatory Transformercited by 2
  25. 2024Efficient generative adversarial networks using linear additive-attention Transformerscited by 2
  26. 2024CrossGP: Cross-Day Glucose Prediction Excluding Physiological Informationcited by 2
  27. 2024Linear-Complexity Self-Supervised Learning for Speech Processingcited by 2
  28. 2024An Analysis of Linear Complexity Attention Substitutes with BEST-RQcited by 2
  29. 2022MMTSA: Multimodal Temporal Segment Attention Network for Efficient Human Activity Recognitioncited by 1
  30. 2024Toward Short-Term Glucose Prediction Solely Based on CGM Time Seriescited by 1
  31. 2024Breaking the Attention Bottleneckcited by 1
  32. 2024Linear Time Complexity Conformers with SummaryMixing for Streaming Speech Recognitioncited by 1
  33. 2024Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Modelscited by 1
  34. 2025A Survey on LLM-based News Recommender Systemscited by 1
  35. 2025Personalized News Recommendation with Multi-granularity Candidate-aware User Modelingcited by 1
  36. 2025Similarity-Aware Selective State-Space Modeling for Semantic Correspondencecited by 1
  37. 2022Fair Comparison between Efficient Attentionscited by 0
  38. 2022Rethinking Query-Key Pairwise Interactions in Vision Transformerscited by 0

50 most cited of 61 All 61 (JSON)