mlsanityData to 7 Oct 2026

LongNet: Scaling Transformers to 1,000,000,000 Tokens

arXiv 2307.02486 ↗First posted 5 Jul 2023cs.CL cs.LGcited by 201

Not on the map. Why?

Abstract (arXiv)

Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. To address this issue, we introduce LongNet, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences. Specifically, we propose dilated attention, which expands the attentive field exponentially as the distance grows. LongNet has significant advantages: 1) it has a linear computation complexity and a logarithm dependency between any two tokens in a sequence; 2) it can be served as a distributed trainer for extremely long sequences; 3) its dilated attention is a drop-in replacement for standard attention, which can be seamlessly integrated with the existing Transformer-based optimization. Experiments results demonstrate that LongNet yields strong performance on both long-sequence modeling and general language tasks. Our work opens up new possibilities for modeling very long sequences, e.g., treating a whole corpus or even the entire Internet as a sequence.

Key facts. Cited by 201 papers in the map (data through 7 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

43 of 53 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
31 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2020Rethinking Attention with Performerscited by 1,584
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2021Scaling Vision Transformerscited by 993
  11. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  12. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  13. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  14. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  15. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  16. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  17. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  18. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  19. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  20. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  21. 2022Memorizing Transformerscited by 219
  22. 2023Augmenting Language Models with Long-Term Memorycited by 186
  23. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  24. 2022A Length-Extrapolatable Transformercited by 123
  25. 2022The Devil in Linear Transformercited by 98
  26. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  27. 2021Luna: Linear Unified Nested Attentioncited by 96
  28. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  29. 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
  30. 2022Foundation Transformerscited by 19
  31. 2022TorchScale: Transformers at Scalecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 33; 2024: 72; 2025: 58; 2026: 38.

201 papers.

Sort:

By area:

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  6. 2023A Comprehensive Overview of Large Language Modelscited by 639
  7. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  8. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  9. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  10. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  11. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  12. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
38 more
  1. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  2. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  3. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  4. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  5. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  6. 2023Efficient Large Language Models: A Surveycited by 183
  7. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  8. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  9. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  10. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  11. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  12. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  13. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 100
  14. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 97
  15. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 94
  16. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 89
  17. 2024Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scalecited by 87
  18. 2023HyperAttention: Long-context Attention in Near-Linear Timecited by 85
  19. 2024PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathologycited by 80
  20. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  21. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 74
  22. 2023Thread of Thought Unraveling Chaotic Contextscited by 71
  23. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  24. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  25. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  26. 2024A Clinical Benchmark of Public Self-Supervised Pathology Foundation Modelscited by 64
  27. 2024A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Modelcited by 59
  28. 2024SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understandingcited by 54
  29. 2023Large Language Models for Compiler Optimizationcited by 52
  30. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  31. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  32. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  33. 2023Extending Context Window of Large Language Models via Semantic Compressioncited by 45
  34. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  35. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43
  36. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
  37. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  38. 2024Meta Large Language Model Compiler: Foundation Models of Compiler Optimizationcited by 39

50 most cited of 201 All 201 (JSON)