mlsanityData to 7 Oct 2026

Efficient Attention via Control Variates

arXiv 2302.04542 ↗First posted 9 Feb 2023cs.LG cs.CL cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Random-feature-based attention (RFA) is an efficient approximation of softmax attention with linear runtime and space complexity. However, the approximation gap between RFA and conventional softmax attention is not well studied. Built upon previous progress of RFA, we characterize this gap through the lens of control variates and show that RFA can be decomposed into a sum of multiple control variate estimators for each element in the sequence. This new framework reveals that exact softmax attention can be recovered from RFA by manipulating each control variate. Besides, it allows us to develop a more flexible form of control variates, resulting in a novel attention mechanism that significantly reduces the approximation gap while maintaining linear complexity. Extensive experiments demonstrate that our model outperforms state-of-the-art efficient attention mechanisms on both vision and language tasks.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 83 of its 103 references that are in the map. First posted on arXiv 9 Feb 2023.

Builds on

83 of 103 references.

References · 83

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2016Layer Normalizationcited by 5,579
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2017Deep Setscited by 1,394
  13. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  14. 2020Designing Network Design Spacescited by 954
  15. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  16. 2018Image Transformercited by 770
  17. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  18. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  19. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  20. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  21. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  22. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  23. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  24. 2021Early Convolutions Help Transformers See Bettercited by 395
  25. 2021Random Feature Attentioncited by 350
  26. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  27. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  28. 2013Black Box Variational Inferencecited by 317
  29. 2020Sparse Sinkhorn Attentioncited by 250
  30. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  31. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  32. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  33. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  34. 2018Scaling Neural Machine Translationcited by 204
  35. 2022Transformer Quality in Linear Timecited by 189
  36. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  37. 2018Online normalizer calculation for softmaxcited by 154
  38. 2021Self-attention Does Not Need O(n2) Memorycited by 152

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 10; 2025: 4; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024Many-Shot In-Context Learningcited by 203
  2. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  3. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  4. 2023TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormercited by 34
  5. 2024Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attentioncited by 31
  6. 2023SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attentioncited by 20
  7. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 20
  8. 2024DiJiang: Efficient Large Language Models through Compact Kernelizationcited by 15
  9. 2024Linear Attention Sequence Parallelismcited by 8
  10. 2024TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmaxcited by 6
  11. 2023Linear Attention via Orthogonal Memorycited by 4
  12. 2023FLuRKA: Fast and accurate unified Low-Rank & Kernel Attentioncited by 3
11 more
  1. 2025Scaling Reasoning without Attentioncited by 3
  2. 2026Proxy Compression for Language Modelingcited by 3
  3. 2024FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Modelscited by 2
  4. 2023Coneheads: Hierarchy Aware Attentioncited by 1
  5. 2024Interactive Multi-Head Self-Attention with Linear Complexitycited by 1
  6. 2025FlashEVA: Accelerating LLM inference via Efficient Attentioncited by 1
  7. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
  8. 2025Feature Shift Localization Networkcited by 0
  9. 2025Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Feature Learning by Learnable Channel Attentioncited by 0
  10. 2026Data-Aware Random Feature Kernel for Transformerscited by 0
  11. 2026Temporal Preservation over Processing: Diagnosing and Designing Spatiotemporal Single-Stage Video Detectorscited by 0