mlsanityData to 7 Oct 2026

Luna: Linear Unified Nested Attention

arXiv 2106.01540 ↗First posted 3 Jun 2021cs.LG cs.CLcited by 96

Not on the map. Why?

Abstract (arXiv)

The quadratic computational and memory complexities of the Transformer's attention mechanism have limited its scalability for modeling long sequences. In this paper, we propose Luna, a linear unified nested attention mechanism that approximates softmax attention with two nested linear attention functions, yielding only linear (as opposed to quadratic) time and space complexity. Specifically, with the first attention function, Luna packs the input sequence into a sequence of fixed length. Then, the packed sequence is unpacked using the second attention function. As compared to a more traditional attention mechanism, Luna introduces an additional sequence with a fixed length as input and an additional corresponding output, which allows Luna to perform attention operation linearly, while also storing adequate contextual information. We perform extensive evaluations on three benchmarks of sequence modeling tasks: long-context sequence modeling, neural machine translation and masked language modeling for large-scale pretraining. Competitive or even better experimental results demonstrate both the effectiveness and efficiency of Luna compared to a variety

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

45 of 59 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016Layer Normalizationcited by 5,579
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
33 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2020On Layer Normalization in the Transformer Architecturecited by 906
  11. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  12. 2020Efficient Transformers: A Surveycited by 784
  13. 2018Image Transformercited by 770
  14. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  15. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  16. 2018Universal Transformerscited by 613
  17. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  18. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  19. 2021Random Feature Attentioncited by 350
  20. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  21. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  22. 2020Sparse Sinkhorn Attentioncited by 250
  23. 2018A Simple Method for Commonsense Reasoningcited by 223
  24. 2018Scaling Neural Machine Translationcited by 204
  25. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  26. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 129
  27. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  28. 2020ProGen: Language Modeling for Protein Generationcited by 127
  29. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
  30. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  31. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 83
  32. 2020GMAT: Global Memory Augmentation for Transformerscited by 36
  33. 2020Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimizationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 20; 2023: 27; 2024: 25; 2025: 8; 2026: 6.

96 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  4. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  5. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  6. 2021A Survey of Transformerscited by 399
  7. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  8. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  9. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  10. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  11. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  12. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
38 more
  1. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  2. 2022Liquid Structural State-Space Modelscited by 134
  3. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  4. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  5. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  6. 2023Diffusion Models Without Attentioncited by 75
  7. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  8. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  9. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  10. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  11. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  12. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  13. 2021Joint Global and Local Hierarchical Priors for Learned Image Compressioncited by 52
  14. 2024Linearizing Large Language Modelscited by 51
  15. 2023A Survey on Long Text Modeling with Transformerscited by 47
  16. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  17. 2023Adaptive Frequency Filters As Efficient Global Token Mixerscited by 45
  18. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  19. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  20. 2021ABC: Attention with Bounded-memory Controlcited by 36
  21. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
  22. 2021Anchor DETR: Query Design for Transformer-Based Object Detectioncited by 30
  23. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  24. 2023Making Vision Transformers Efficient from A Token Sparsification Viewcited by 28
  25. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  26. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  27. 2023RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generationcited by 25
  28. 2023Efficient Attention via Control Variatescited by 23
  29. 2023Object Detection with Transformers: A Reviewcited by 22
  30. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  31. 2023Sparse Modular Activation for Efficient Sequence Modelingcited by 20
  32. 2024The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?cited by 20
  33. 2023Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operatorcited by 19
  34. 2024CascadedGaze: Efficiency in Global Context Extraction for Image Restorationcited by 16
  35. 2021On Learning the Transformer Kernelcited by 14
  36. 2022Attention-based Random Forest and Contamination Modelcited by 10
  37. 2022A Call for Clarity in Beam Search: How It Works and When It Stopscited by 10
  38. 2023On the Long Range Abilities of Transformerscited by 10

50 most cited of 96 All 96 (JSON)