mlsanityData to 7 Oct 2026

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory

arXiv 2402.04617 ↗First posted 7 Feb 2024cs.CL cs.AI cs.LGcited by 192

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

Large language models (LLMs) have emerged as a cornerstone in real-world applications with lengthy streaming inputs (e.g., LLM-driven agents). However, existing LLMs, pre-trained on sequences with a restricted maximum length, cannot process longer sequences due to the out-of-domain and distraction issues. Common solutions often involve continual pre-training on longer sequences, which will introduce expensive computational overhead and uncontrollable change in model capabilities. In this paper, we unveil the intrinsic capacity of LLMs for understanding extremely long sequences without any fine-tuning. To this end, we introduce a training-free memory-based method, InfLLM. Specifically, InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention computation. Thereby, InfLLM allows LLMs to efficiently process long sequences with a limited context window and well capture long-distance dependencies. Without any training, InfLLM enables LLMs that are pre-trained on sequences consisting of a few thousand tokens to achieve comparable performance with competitive baselines that continually train these LLMs on long sequences. Even when the sequence length is scaled to 1,024K, InfLLM still effectively captures long-distance dependencies. Our code can be found in urlhttps://github.com/thunlp/InfLLM.

Key facts. Cited by 192 papers in the map (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 54 of its 71 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

54 of 71 references.

References · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  12. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  13. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  14. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  15. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  16. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  17. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  18. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  19. 2020Efficient Transformers: A Surveycited by 784
  20. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  21. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  22. 2014Neural Turing Machinescited by 630
  23. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  24. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  25. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  26. 2014Memory Networkscited by 354
  27. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  28. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  29. 2023Improving Text Embeddings with Large Language Modelscited by 295
  30. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  31. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  32. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  33. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  34. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  35. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  36. 2022A Length-Extrapolatable Transformercited by 123
  37. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  38. 2023Retrieval meets Long Context Large Language Modelscited by 114

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 78; 2026: 94.

192 papers.

Sort:

By area:

  1. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  2. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  3. 2024Titans: Learning to Memorize at Test Timecited by 303
  4. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  5. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  6. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  7. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  8. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  9. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  10. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  11. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  12. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
38 more
  1. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 55
  2. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  3. 2024SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separatorcited by 41
  4. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 38
  5. 2025Context as a Tool: Context Management for Long-Horizon SWE-Agentscited by 38
  6. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  7. 2026MiniMax Sparse Attentioncited by 34
  8. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  9. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  10. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  11. 2024Emotional RAG: Enhancing Role-Playing Agents through Emotional Retrievalcited by 26
  12. 2024Star Attention: Efficient LLM Inference over Long Sequencescited by 25
  13. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  14. 2025SageAttention2++: A More Efficient Implementation of SageAttention2cited by 21
  15. 2024SCOPE: Optimizing Key-Value Cache Compression in Long-context Generationcited by 20
  16. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  17. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inferencecited by 18
  18. 2024LLM×MapReduce: Simplified Long-Sequence Processing using Large Language Modelscited by 17
  19. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  20. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  21. 2024On the token distance modeling ability of higher RoPE attention dimensioncited by 13
  22. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 13
  23. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  24. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruningcited by 13
  25. 2025AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inferencecited by 12
  26. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 12
  27. 2024LongSafety: Enhance Safety for Long-Context LLMscited by 11
  28. 2025Inference-time sparse attention with asymmetric indexingcited by 11
  29. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimizationcited by 10
  30. 2025Rectified Sparse Attentioncited by 10
  31. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  32. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Cachingcited by 9
  33. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  34. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9
  35. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decodingcited by 9
  36. 2026Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generationcited by 9
  37. 2025LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Informationcited by 8
  38. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMscited by 8

50 most cited of 192 All 192 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
  2. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  3. 2026Summarize Before You Speak with ARACH: A Training-Free Inference-Time Plug-In for Enhancing LLMs via Global Attention Reallocation
  4. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts
  5. 2023SparQ Attention: Bandwidth-Efficient LLM Inference
  6. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts
  7. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
  8. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline