mlsanityData to 7 Oct 2026

Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models

arXiv 2402.02244 ↗First posted 3 Feb 2024cs.CL cs.LGcited by 71

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recently, large language models (LLMs) have shown remarkable capabilities including understanding context, engaging in logical reasoning, and generating responses. However, this is achieved at the expense of stringent computational and memory requirements, hindering their ability to effectively support long input sequences. This survey provides an inclusive review of the recent techniques and methods devised to extend the sequence length in LLMs, thereby enhancing their capacity for long-context understanding. In particular, we review and categorize a wide range of techniques including architectural modifications, such as modified positional encoding and altered attention mechanisms, which are designed to enhance the processing of longer sequences while avoiding a proportional increase in computational requirements. The diverse methodologies investigated in this study can be leveraged across different phases of LLMs, i.e., training, fine-tuning and inference. This enables LLMs to efficiently process extended sequences. The limitations of the current methodologies is discussed in the last section along with the suggestions for future research directions, underscoring the importance of sequence length in the continued advancement of LLMs.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 72 of its 79 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

72 of 79 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023PaLM 2 Technical Report17 May
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
38 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  12. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  13. 2023StarCoder: may the source be with you!cited by 790
  14. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  15. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  16. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  17. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  18. 2023A Comprehensive Overview of Large Language Modelscited by 639
  19. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  20. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  21. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  22. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  23. 2022Efficiently Scaling Transformer Inferencecited by 529
  24. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  25. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  26. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  27. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  28. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  29. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  30. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  31. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  32. 2023A Survey on Model Compression for Large Language Modelscited by 263
  33. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  34. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  35. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  36. 2023Efficient Large Language Models: A Surveycited by 183
  37. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  38. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 39; 2026: 14.

71 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  3. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  4. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 108
  5. 2025Cameras as Relative Positional Encodingcited by 94
  6. 2024Timer-XL: Long-Context Transformers for Unified Time Series Forecastingcited by 83
  7. 2024Self-Evolving Multi-Agent Collaboration Networks for Software Developmentcited by 73
  8. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  9. 2025EnvBench: A Benchmark for Automated Environment Setupcited by 36
  10. 2025Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrativecited by 35
  11. 2024TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attentioncited by 28
  12. 2024Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentationcited by 24
38 more
  1. 2024Self-Updatable Large Language Models by Integrating Context into Model Parameterscited by 21
  2. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  3. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  4. 2024GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understandingcited by 18
  5. 2025FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generationcited by 17
  6. 2025Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilizationcited by 15
  7. 2024Towards LifeSpan Cognitive Systemscited by 13
  8. 2025From Reproduction to Replication: Evaluating Research Agents with Progressive Code Maskingcited by 12
  9. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  10. 2024Q-S5: Towards Quantized State Space Modelscited by 9
  11. 2025Procedural Memory Is Not All You Need: Bridging Cognitive Gaps in LLM-Based Agentscited by 9
  12. 2025Evaluating the Sensitivity of LLMs to Prior Contextcited by 9
  13. 2025Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architecturescited by 9
  14. 2025An Empirical Study on Prompt Compression for Large Language Modelscited by 8
  15. 2025AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Sciencecited by 7
  16. 2025FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernelcited by 7
  17. 2025Fundamentals of Building Autonomous LLM Agentscited by 7
  18. 2024Towards Ethical Personal AI Applications: Practical Considerations for AI Assistants with Long-Term Memorycited by 6
  19. 2025Everything is Context: Agentic File System Abstraction for Context Engineeringcited by 6
  20. 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrievalcited by 5
  21. 2025KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoningcited by 5
  22. 2025SelfCheck-Eval: A Multi-Module Framework for Zero-Resource Hallucination Detection in Large Language Modelscited by 4
  23. 2025Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunitiescited by 4
  24. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  25. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contextscited by 4
  26. 2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Servingcited by 3
  27. 2025EVADE-Bench: Multimodal Benchmark for Evaluating and Enhancing Evasive Content Detectioncited by 3
  28. 2025T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generationcited by 3
  29. 2025Court of LLMs: Evidence-Augmented Generation via Multi-LLM Collaboration for Text-Attributed Graph Anomaly Detectioncited by 3
  30. 2026The Y-Combinator for LLMs: Solving Long-Context Rot with λ-Calculuscited by 3
  31. 2024CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novelscited by 2
  32. 2025Top-Theta Attention: Sparsifying Transformers by Compensated Thresholdingcited by 2
  33. 2026Scaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspacescited by 2
  34. 2025PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inferencecited by 1
  35. 2025Direct Advantage Regression: Aligning LLMs with Online AI Rewardcited by 1
  36. 2025LLMSynthor: Macro-Aligned Micro-Records Synthesis with Large Language Modelscited by 1
  37. 2025Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluationcited by 1
  38. 2025MeVe: A Modular System for Memory Verification and Effective Context Control in Language Modelscited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  2. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMs
  3. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
  4. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
  5. 2024Large Language Models Can Self-Improve in Long-context Reasoning
  6. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
  7. 2024LongSSM: On the Length Extension of State-space Models in Language Modelling
  8. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning