mlsanityData to 14 Oct 2026

The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey

arXiv 2401.07872 ↗First posted 15 Jan 2024cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

The advent of Large Language Models (LLMs) represents a notable breakthrough in Natural Language Processing (NLP), contributing to substantial progress in both text comprehension and generation. However, amidst these advancements, it is noteworthy that LLMs often face a limitation in terms of context length extrapolation. Understanding and extending the context length for LLMs is crucial in enhancing their performance across various NLP applications. In this survey paper, we delve into the multifaceted aspects of exploring why it is essential, and the potential transformations that superior techniques could bring to NLP applications. We study the inherent challenges associated with extending context length and present an organized overview of the existing strategies employed by researchers. Additionally, we discuss the intricacies of evaluating context extension techniques and highlight the open challenges that researchers face in this domain. Furthermore, we explore whether there is a consensus within the research community regarding evaluation standards and identify areas where further agreement is needed. This comprehensive survey aims to serve as a valuable resource for researchers, guiding them through the nuances of context length extension techniques and fostering discussions on future advancements in this evolving field.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 114 of its 158 references that are in the map. First posted on arXiv 15 Jan 2024.

Builds on

114 of 158 references.

References · 114

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  11. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  12. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
38 more
  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  13. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  14. 2020Longformer: The Long-Document Transformercited by 3,413
  15. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  16. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  17. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  18. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  19. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  20. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  21. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  22. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  23. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  24. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  25. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  26. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  27. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  28. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  29. 2020Reformer: The Efficient Transformercited by 1,668
  30. 2020Rethinking Attention with Performerscited by 1,601
  31. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  32. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  33. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  34. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  35. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  36. 2018Self-Attention with Relative Position Representationscited by 1,120
  37. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  38. 2022Token Merging: Your ViT But Fastercited by 1,021

50 most cited of 114 All 114 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 15; 2026: 3.

33 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  3. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  4. 2024With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generationcited by 37
  5. 2024Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculumcited by 26
  6. 2024Language Models can Self-Lengthen to Generate Long Textscited by 24
  7. 2024Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learningcited by 23
  8. 2025Mid-Training of Large Language Models: A Surveycited by 21
  9. 2024TULIP: Token-length Upgraded CLIPcited by 19
  10. 2024Exploring Context Window of Large Language Models via Decomposed Positional Vectorscited by 18
  11. 2024Compressing Long Context for Enhancing RAG with AMR-based Concept Distillationcited by 16
  12. 2025A Cognitive Writing Perspective for Constrained Long-Form Text Generationcited by 14
21 more
  1. 2025An LLM-Based Approach for Insight Generation in Data Analysiscited by 12
  2. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 12
  3. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  4. 2025Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Servicescited by 10
  5. 2025CAIM: Development and Evaluation of a Cognitive AI Memory Framework for Long-Term Interaction with Intelligent Agentscited by 7
  6. 2024ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Modelscited by 6
  7. 2024Self-Attention Limits Working Memory Capacity of Transformer-Based Modelscited by 5
  8. 2025CoRank: LLM-Based Compact Reranking with Document Features for Scientific Retrievalcited by 5
  9. 2025Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attentioncited by 5
  10. 2025MOPrompt: Multi-objective Semantic Evolution for Prompt Optimizationcited by 4
  11. 2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Servingcited by 3
  12. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 3
  13. 2024CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novelscited by 2
  14. 2025An Empirical Study of OpenAI API Discussions on Stack Overflowcited by 1
  15. 2025AI Compute Architecture and Evolution Trendscited by 1
  16. 2024A Study on Context Length and Efficient Transformers for Biomedical Image Analysiscited by 0
  17. 2025Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generationcited by 0
  18. 2025Wireless Power Transfer and Intent-Driven Network Optimization in AAVs-assisted IoT for 6G Sustainable Connectivitycited by 0
  19. 2026Towards Infinite Length Extrapolation: A Unified Approachcited by 0
  20. 2026Context-Length Robustness in Question Answering Models: A Comparative Empirical Studycited by 0
  21. 2026Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devicescited by 0