mlsanityData to 7 Oct 2026

Rethinking Attention with Performers

arXiv 2009.14794 ↗First posted 30 Sep 2020cs.LG cs.CL stat.MLcited by 1,584

Not on the map. Why?

Cited by 61 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attention-kernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can be also used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.

Key facts. Cited by 1,584 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 185; 2022: 262; 2023: 249; 2024: 297; 2025: 314; 2026: 277.

1,584 papers.

Sort:

By area:

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  7. 2021ViViT: A Video Vision Transformercited by 1,610
  8. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  9. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  11. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  12. 2021Transformers in Vision: A Surveycited by 1,009
38 more
  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  3. 2021Multiscale Vision Transformerscited by 896
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  5. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  6. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  7. 2022Transformers learn in-context by gradient descentcited by 714
  8. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  9. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  10. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  11. 2022Efficiently Scaling Transformer Inferencecited by 529
  12. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  13. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  14. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  15. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  16. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  17. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  18. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  19. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  20. 2021Neural Operator: Learning Maps Between Function Spacescited by 413
  21. 2021A Survey of Transformerscited by 399
  22. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  23. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  24. 2021Random Feature Attentioncited by 350
  25. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  26. 2023Adapting Language Models to Compress Contextscited by 350
  27. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  28. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  29. 2021Choose a Transformer: Fourier or Galerkincited by 333
  30. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  31. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  32. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  33. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  34. 2024Titans: Learning to Memorize at Test Timecited by 303
  35. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 300
  36. 2024Transolver: A Fast Transformer Solver for PDEs on General Geometriescited by 299
  37. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  38. 2022Long Range Language Modeling via Gated State Spacescited by 265

50 most cited of 1,584 All 1,584 (JSON)