mlsanityData to 7 Oct 2026

Linear Log-Normal Attention with Unbiased Concentration

arXiv 2311.13541 ↗First posted 22 Nov 2023cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-attention mechanism concerning the sequence length. This limitation poses a substantial obstacle when dealing with long documents or high-resolution images. In this work, we study the self-attention mechanism by analyzing the distribution of the attention matrix and its concentration ability. Furthermore, we propose instruments to measure these quantities and introduce a novel self-attention mechanism, Linear Log-Normal Attention, designed to emulate the distribution and concentration behavior of the original self-attention. Our experimental results on popular natural language benchmarks reveal that our proposed Linear Log-Normal Attention outperforms other linearized attention alternatives, offering a promising avenue for enhancing the scalability of transformer models.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 41 of its 59 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

41 of 59 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
29 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  10. 2017Deep Neural Networks as Gaussian Processescited by 617
  11. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  12. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  13. 2019Axial Attention in Multidimensional Transformerscited by 350
  14. 2021Random Feature Attentioncited by 350
  15. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  16. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  17. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  18. 2020Fast Transformers with Clustered Attentioncited by 123
  19. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
  20. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  21. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  22. 2019On Extractive and Abstractive Neural Document Summarization with Transformer Language Modelscited by 83
  23. 2021KVT: k-NN Attention for Boosting Vision Transformerscited by 64
  24. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 61
  25. 2023Toeplitz Neural Network for Sequence Modelingcited by 43
  26. 2018Post-training 4-bit quantization of convolution networks for rapid-deploymentcited by 38
  27. 2018An Analysis of Neural Language Modeling at Multiple Scalescited by 31
  28. 2017What does Attention in Neural Machine Translation Pay Attention to?cited by 30
  29. 2021Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machinescited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 4; 2026: 2.

11 papers.

  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  3. 2024Length-Induced Embedding Collapse in PLM-based Modelscited by 9
  4. 2025ReGLA: Refining Gated Linear Attentioncited by 4
  5. 2024ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Modelscited by 3
  6. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  7. 2025Top-Theta Attention: Sparsifying Transformers by Compensated Thresholdingcited by 2
  8. 2025Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attentioncited by 2
  9. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 2
  10. 2025Power Law Guided Dynamic Sifting for Efficient Attentioncited by 1
  11. 2026MirrorLA: Reflecting Feature Map for Vision Linear Attentioncited by 0