mlsanityData to 7 Oct 2026

Random Feature Attention

arXiv 2103.02143 ↗First posted 3 Mar 2021cs.CLcited by 350

Not on the map. Why?

Abstract (arXiv)

Transformers are state-of-the-art models for a variety of sequence modeling tasks. At their core is an attention function which models pairwise interactions between the inputs at every timestep. While attention is powerful, it does not scale efficiently to long sequences due to its quadratic time and space complexity in the sequence length. We propose RFA, a linear time and space attention that uses random feature methods to approximate the softmax function, and explore its application in transformers. RFA can be used as a drop-in replacement for conventional softmax attention and offers a straightforward way of learning with recency bias through an optional gating mechanism. Experiments on language modeling and machine translation demonstrate that RFA achieves similar or better performance compared to strong transformer baselines. In the machine translation experiment, RFA decodes twice as fast as a vanilla transformer. Compared to existing efficient transformer variants, RFA is competitive in terms of both accuracy and efficiency on three long text classification datasets. Our analysis shows that RFA's efficiency gains are especially notable on long sequences, suggesting that RFA will be particularly useful in tasks that require working with large inputs, fast decoding speed, or low memory footprints.

Key facts. Cited by 350 papers in the map (data through 7 Oct 2026). Builds on 59 of its 79 references that are in the map. First posted on arXiv 3 Mar 2021.

Builds on

59 of 79 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  10. 2013Auto-Encoding Variational Bayescited by 3,980
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
38 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  11. 2020Efficient Transformers: A Surveycited by 784
  12. 2018Image Transformercited by 770
  13. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  14. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  15. 2018Universal Transformerscited by 613
  16. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  17. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  18. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  19. 2019Axial Attention in Multidimensional Transformerscited by 350
  20. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  21. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  22. 2020Sparse Sinkhorn Attentioncited by 250
  23. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  24. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  25. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  26. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  27. 2018Scaling Neural Machine Translationcited by 204
  28. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  29. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  30. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  31. 2019Adaptive Attention Span in Transformerscited by 148
  32. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  33. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  34. 2016Orthogonal Random Featurescited by 89
  35. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  36. 2018Differentiable plasticity: training plastic neural networks with backpropagationcited by 83
  37. 2017Classical Structured Prediction Losses for Sequence to Sequence Learningcited by 55
  38. 2019Transformers with convolutional context for ASRcited by 44

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 49; 2022: 75; 2023: 66; 2024: 67; 2025: 48; 2026: 45.

350 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  4. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  5. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  6. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  7. 2021A Survey of Transformerscited by 399
  8. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  9. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  10. 2021Choose a Transformer: Fourier or Galerkincited by 333
  11. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  12. 2021Pre-Trained Models: Past, Present and Futurecited by 319
38 more
  1. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  2. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  3. 2022Long Range Language Modeling via Gated State Spacescited by 265
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  5. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  6. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  7. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  8. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  9. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  10. 2022Transformer Quality in Linear Timecited by 189
  11. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  12. 2023Efficient Large Language Models: A Surveycited by 183
  13. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  14. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  15. 2023Spike-driven Transformercited by 141
  16. 2022On The Computational Complexity of Self-Attentioncited by 137
  17. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  18. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  19. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  20. 2024Transformers are Multi-State RNNscited by 120
  21. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  22. 2022Training Language Models with Memory Augmentationcited by 112
  23. 2021An Attention Free Transformercited by 111
  24. 2022Block-Recurrent Transformerscited by 107
  25. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  26. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  27. 2022The Devil in Linear Transformercited by 98
  28. 2021Luna: Linear Unified Nested Attentioncited by 96
  29. 2021Finetuning Pretrained Transformers into RNNscited by 94
  30. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  31. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  32. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  33. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  34. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
  35. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 84
  36. 2022Fast Vision Transformers with HiLo Attentioncited by 78
  37. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  38. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75

50 most cited of 350 All 350 (JSON)