mlsanityData to 7 Oct 2026

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

arXiv 2401.01325 ↗First posted 2 Jan 2024cs.CL cs.AI cs.LGcited by 133

Not on the map. Why?

Abstract (arXiv)

It is well known that LLMs cannot generalize well to long contexts whose lengths are larger than the training sequence length. This poses challenges when employing LLMs for processing long input sequences during inference. In this work, we argue that LLMs themselves have inherent capabilities to handle long contexts without fine-tuning. To achieve this goal, we propose SelfExtend to extend the context window of LLMs by constructing bi-level attention information: the grouped attention and the neighbor attention. The grouped attention captures the dependencies among tokens that are far apart, while neighbor attention captures dependencies among adjacent tokens within a specified range. The two-level attentions are computed based on the original model's self-attention mechanism during inference. With minor code modification, our SelfExtend can effortlessly extend existing LLMs' context window without any fine-tuning. We conduct comprehensive experiments on multiple benchmarks and the results show that our SelfExtend can effectively extend existing LLMs' context window length. The code can be found at urlhttps://github.com/datamllab/LongLM.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 2 Jan 2024.

Builds on

43 of 55 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2023Mistral 7Bcited by 3,065
31 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  11. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  13. 2023Instruction Tuning with GPT-4cited by 676
  14. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  15. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  16. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  17. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  18. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  19. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  20. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  21. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  22. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  23. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  24. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  25. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  26. 2022A Length-Extrapolatable Transformercited by 123
  27. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  28. 2021SparseBERT: Rethinking the Importance Analysis in Self-attentioncited by 46
  29. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43
  30. 2023Giraffe: Adventures in Expanding Context Lengths in LLMscited by 32
  31. 2022When Neural Networks Fail to Generalize? A Model Sensitivity Perspectivecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 60; 2025: 52; 2026: 21.

133 papers.

Sort:

By area:

  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  2. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  3. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  4. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  5. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  6. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  7. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  8. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 118
  9. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 104
  10. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  11. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  12. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
38 more
  1. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 93
  2. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 74
  3. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  4. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  5. 2024Length Generalization of Causal Transformers without Position Encodingcited by 46
  6. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  7. 2024Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Modelscited by 41
  8. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaborationcited by 38
  9. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 37
  10. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 34
  11. 2024Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformerscited by 32
  12. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  13. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 31
  14. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  15. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 29
  16. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  17. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 25
  18. 2024Unified Parameter-Efficient Unlearning for LLMscited by 24
  19. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
  20. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  21. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  22. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  23. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 19
  24. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  25. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  26. 2024Exploring Context Window of Large Language Models via Decomposed Positional Vectorscited by 18
  27. 2024LoCoCo: Dropping In Convolutions for Long Context Compressioncited by 18
  28. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  29. 2024LongHeads: Multi-Head Attention is Secretly a Long Context Processorcited by 16
  30. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 16
  31. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  32. 2024What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practicescited by 14
  33. 2024MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokenscited by 13
  34. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  35. 2024Agentic Retrieval-Augmented Generation for Time Series Analysiscited by 13
  36. 2024Towards LifeSpan Cognitive Systemscited by 13
  37. 2024On the token distance modeling ability of higher RoPE attention dimensioncited by 13
  38. 2025MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learningcited by 13

50 most cited of 133 All 133 (JSON)