mlsanityData to 7 Oct 2026

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

arXiv 2404.07143 ↗First posted 10 Apr 2024cs.CL cs.AI cs.LGcited by 194

Not on the map. Why?

Cited by 32 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Builds on 48 of its 64 references that are in the map. First posted on arXiv 10 Apr 2024.

Builds on

48 of 64 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
36 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  6. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  8. 2014Neural Turing Machinescited by 630
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  10. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  11. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  13. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  14. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  15. 2023Adapting Language Models to Compress Contextscited by 350
  16. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  17. 2017Meta Networkscited by 268
  18. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  19. 2022Memorizing Transformerscited by 219
  20. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  21. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  22. 2022Recurrent Memory Transformercited by 197
  23. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  24. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  25. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  26. 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
  27. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  28. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  29. 2015Neural GPUs Learn Algorithmscited by 118
  30. 2018Differentiable plasticity: training plastic neural networks with backpropagationcited by 83
  31. 2021PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarizationcited by 62
  32. 2020Learning Associative Inference Using Fast Weight Memorycited by 51
  33. 2019Metalearned Neural Memorycited by 44
  34. 2019Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solvingcited by 34
  35. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  36. 2016Neural Semantic Encoderscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 64; 2025: 67; 2026: 63.

194 papers.

Sort:

By area:

  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  3. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  4. 2024Titans: Learning to Memorize at Test Timecited by 303
  5. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  6. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  7. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  8. 2025MIRIX: Multi-Agent Memory System for LLM-Based Agentscited by 160
  9. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  10. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  11. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  12. 2025Test-Time Training Done Rightcited by 121
38 more
  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  3. 2025Recursive Language Modelscited by 94
  4. 2024WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Modelscited by 90
  5. 2025Efficient Reasoning with Hidden Thinkingcited by 72
  6. 2025A Survey on Latent Reasoningcited by 71
  7. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  8. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  9. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  10. 2024Linearizing Large Language Modelscited by 51
  11. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  12. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  13. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 42
  14. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  15. 2025Log-Linear Attentioncited by 39
  16. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 38
  17. 2025VRAG: Learning World Models for Interactive Video Generationcited by 37
  18. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  19. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  20. 2024Needle in the Haystack for Memory Based Large Language Modelscited by 32
  21. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agentscited by 31
  22. 2024Selective Attention Improves Transformercited by 26
  23. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 25
  24. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25
  25. 2024Context Parallelism for Scalable Million-Token Inferencecited by 25
  26. 2024Star Attention: Efficient LLM Inference over Long Sequencescited by 25
  27. 2024TransformerFAM: Feedback attention is working memorycited by 21
  28. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  29. 2024An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encodingcited by 16
  30. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  31. 2024ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attentioncited by 13
  32. 2024Towards LifeSpan Cognitive Systemscited by 13
  33. 2024Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Modelscited by 13
  34. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMscited by 13
  35. 2024Beyond Scaling Laws: Understanding Transformer Performance with Associative Memorycited by 12
  36. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  37. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  38. 2026Memory Caching: RNNs with Growing Memorycited by 12

50 most cited of 194 All 194 (JSON)