mlsanityData to 7 Oct 2026

LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models

arXiv 2308.16137 ↗First posted 30 Aug 2023cs.CL cs.AIcited by 120

Not on the map. Why?

Cited by 41 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Today's large language models (LLMs) typically train on short text segments (e.g., <4K tokens) due to the quadratic complexity of their Transformer architectures. As a result, their performance suffers drastically on inputs longer than those encountered during training, substantially limiting their applications in real-world tasks involving long contexts such as encoding scientific articles, code repositories, or long dialogues. Through theoretical analysis and empirical investigation, this work identifies three major factors contributing to this length generalization failure. Our theoretical analysis further reveals that commonly used techniques like truncating the attention window or relative positional encodings are inadequate to address them. Answering these challenges, we propose LM-Infinite, a simple and effective method for enhancing LLMs' capabilities of handling long contexts. LM-Infinite is highly flexible and can be used with most modern LLMs off-the-shelf. Without any parameter updates, it allows LLMs pre-trained with 2K or 4K-long segments to generalize to up to 200M length inputs while retaining perplexity. It also improves performance on downstream tasks such as Passkey Retrieval and Qasper in the zero-shot setting. LM-Infinite brings substantial efficiency improvements: it achieves 2.7x decoding speed up and 7.5x memory saving over the original model. Our codes are released at urlhttps://github.com/Glaciohound/LM-Infinite.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Builds on 30 of its 65 references that are in the map. First posted on arXiv 30 Aug 2023.

Builds on

30 of 65 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
18 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  2. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  4. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  5. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  6. 2022Memorizing Transformerscited by 219
  7. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  8. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  9. 2023RecurrentGPT: Interactive Generation of (Arbitrarily) Long Textcited by 84
  10. 2021Adaptive Semiparametric Language Modelscited by 83
  11. 2022Parallel Context Windows for Large Language Modelscited by 81
  12. 2017Learning to Remember Rare Eventscited by 63
  13. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 61
  14. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  15. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 39
  16. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  17. 2021Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequencescited by 5
  18. 2023A Frustratingly Easy Improvement for Position Embeddings via Random Paddingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 27; 2025: 54; 2026: 38.

120 papers.

Sort:

By area:

  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  2. 2024Titans: Learning to Memorize at Test Timecited by 303
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  4. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  5. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  6. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  7. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  8. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  9. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  10. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  11. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  12. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
38 more
  1. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  2. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  3. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  4. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  5. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  6. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  7. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  8. 2026Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Executioncited by 42
  9. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 38
  10. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  11. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruningcited by 26
  12. 2024Star Attention: Efficient LLM Inference over Long Sequencescited by 25
  13. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  14. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Examplecited by 22
  15. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 20
  16. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  17. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  18. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 16
  19. 2024Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistencycited by 14
  20. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  21. 2024Towards LifeSpan Cognitive Systemscited by 13
  22. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 13
  23. 2025LongDiff: Training-Free Long Video Generation in One Gocited by 13
  24. 2025Native-Resolution Image Synthesiscited by 13
  25. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  26. 2025Must Read: A Comprehensive Survey of Computational Persuasioncited by 12
  27. 2025Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skippingcited by 11
  28. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  29. 2026Incantation: Natural Language as the Action Interface for Multi-Entity Video World Modelscited by 10
  30. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Modelscited by 9
  31. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9
  32. 2026Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generationcited by 9
  33. 2025PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attentioncited by 8
  34. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMscited by 8
  35. 2025All You Need is One: Capsule Prompt Tuning with a Single Vectorcited by 8
  36. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  37. 2025InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPUcited by 7
  38. 2025LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Importantcited by 7

50 most cited of 120 All 120 (JSON)