mlsanityData to 14 Oct 2026

Efficient Streaming Language Models with Attention Sinks

arXiv 2309.17453 ↗First posted 29 Sep 2023cs.CL cs.AIcited by 2,186

Not on the map. Why?

Cited by 764 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a "sink" even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.

Key facts. Cited by 2,186 papers in the map (data through 14 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

45 of 62 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
33 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2020Longformer: The Long-Document Transformercited by 3,413
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  11. 2020Reformer: The Efficient Transformercited by 1,668
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  13. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  14. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  15. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  16. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  17. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  18. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  19. 2020Efficient Transformers: A Surveycited by 790
  20. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  21. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  22. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  23. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  24. 2022Efficiently Scaling Transformer Inferencecited by 539
  25. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  26. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  27. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  28. 2021Efficient Attentions for Long Document Summarizationcited by 345
  29. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  30. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  31. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  32. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  33. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 28; 2024: 369; 2025: 675; 2026: 1114.

2,186 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  5. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  6. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  8. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  10. 2023Point Transformer V3: Simpler, Faster, Strongercited by 589
  11. 2024Scaling and evaluating sparse autoencoderscited by 553
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
38 more
  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  2. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  4. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  5. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  6. 2024Titans: Learning to Memorize at Test Timecited by 328
  7. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  8. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  9. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  10. 2024Massive Activations in Large Language Modelscited by 294
  11. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  12. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  13. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  14. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  15. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  16. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  17. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  18. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  19. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  20. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 234
  21. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  22. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  23. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 217
  24. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  25. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  26. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  27. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  28. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  29. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  30. 2025Chronos-2: From Univariate to Universal Forecastingcited by 194
  31. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 192
  32. 2026MiMo-V2-Flash Technical Reportcited by 191
  33. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  34. 2023Efficient Large Language Models: A Surveycited by 186
  35. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  36. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  37. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  38. 2025History-Guided Video Diffusioncited by 171

50 most cited of 2,186 All 2,186 (JSON)