mlsanityData to 7 Oct 2026

Why Does the Effective Context Length of LLMs Fall Short?

arXiv 2410.18745 ↗First posted 24 Oct 2024cs.CLcited by 60

Not on the map. Why?

Abstract (arXiv)

Advancements in distributed training and efficient attention mechanisms have significantly expanded the context window sizes of large language models (LLMs). However, recent work reveals that the effective context lengths of open-source LLMs often fall short, typically not exceeding half of their training lengths. In this work, we attribute this limitation to the left-skewed frequency distribution of relative positions formed in LLMs pretraining and post-training stages, which impedes their ability to effectively gather distant information. To address this challenge, we introduce ShifTed Rotray position embeddING (STRING). STRING shifts well-trained positions to overwrite the original ineffective positions during inference, enhancing performance within their existing training lengths. Experimental results show that without additional training, STRING dramatically improves the performance of the latest large-scale models, such as Llama3.1 70B and Qwen2 72B, by over 10 points on popular long-context benchmarks RULER and InfiniteBench, establishing new state-of-the-art results for open-source LLMs. Compared to commercial models, Llama 3.1 70B with method even achieves better performance than GPT-4-128K and clearly surpasses Claude 2 and Kimi-chat.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 53 of its 70 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

53 of 70 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
38 more
  1. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2024Long Context Transfer from Language to Visioncited by 502
  4. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  6. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  7. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  9. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  10. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  11. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  12. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  13. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  14. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  15. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 139
  16. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  17. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  18. 2024Make Your LLM Fully Utilize the Contextcited by 118
  19. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  20. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  21. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  22. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  23. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  24. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
  25. 2024Base of RoPE Bounds Context Lengthcited by 49
  26. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  27. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  28. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  29. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43
  30. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  31. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  32. 2023DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Trainingcited by 28
  33. 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
  34. 2024Resonance RoPE: Improving Context Length Generalization of Large Language Modelscited by 25
  35. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  36. 2024Long Context Alignment with Short Instructions and Synthesized Positionscited by 22
  37. 2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
  38. 2024Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Modelcited by 15

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 36; 2026: 21.

60 papers.

Sort:

By area:

  1. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  2. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  3. 2025Qwen2.5-1M Technical Reportcited by 158
  4. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  5. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 38
  6. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  7. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 20
  8. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  9. 2025The Future is Agentic: Definitions, Perspectives, and Open Challenges of Multi-Agent Recommender Systemscited by 14
  10. 2026Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answeringcited by 13
  11. 2025R3Mem: Bridging Memory Retention and Retrieval via Reversible Compressioncited by 12
  12. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 11
38 more
  1. 2026Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memorycited by 11
  2. 2025AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Managementcited by 10
  3. 2026GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization (V1.0)cited by 10
  4. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  5. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  6. 2025Measuring How (Not Just Whether) VLMs Build Common Groundcited by 8
  7. 2025SkyLadder: Better and Faster Pretraining via Context Window Schedulingcited by 6
  8. 2025LogiDebrief: A Signal-Temporal Logic based Automated Debriefing Approach with Large Language Models Integrationcited by 5
  9. 2025Revisiting Long-context Modeling from Context Denoising Perspectivecited by 5
  10. 2025Beyond the limitation of a single query: Train your LLM for query expansion with Reinforcement Learningcited by 5
  11. 2026AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generationcited by 5
  12. 2025Actions Speak Louder than Prompts: A Large-Scale Study of LLMs for Graph Inferencecited by 4
  13. 2026End-to-End Context Compression at Scalecited by 4
  14. 2025MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretrainingcited by 3
  15. 2025Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agentscited by 3
  16. 2026Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agentscited by 3
  17. 2024GIRAFFE: Design Choices for Extending the Context Length of Visual Language Modelscited by 2
  18. 2024LABIIUM: AI-Enhanced Zero-configuration Measurement Automation Systemcited by 2
  19. 2025From Haystack to Needle: Label Space Reduction for Zero-shot Classificationcited by 2
  20. 2025SysTemp: A Multi-Agent System for Template-Based Generation of SysML v2cited by 2
  21. 2025Video Parallel Scaling: Aggregating Diverse Frame Subsets for VideoLLMscited by 2
  22. 2026DYCP: Dynamic Context Pruning for Long-Form Dialogue with LLMscited by 2
  23. 2026Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systemscited by 2
  24. 2025Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Modelscited by 1
  25. 2025Respecting Temporal-Causal Consistency: Entity-Event Knowledge Graphs for Retrieval-Augmented Generationcited by 1
  26. 2025LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Trainingcited by 1
  27. 2025MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Trainingcited by 1
  28. 2025"As Eastern Powers, I will veto." : An Investigation of Nation-level Bias of Large Language Models in International Relationscited by 1
  29. 2026Enhancing Reliability in LLM-Based Secure Code Generationcited by 1
  30. 2026Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledgecited by 1
  31. 2025Do Multimodal Large Language Models Understand Welding?cited by 0
  32. 2025ML For Hardware Design Interpretability: Challenges and Opportunitiescited by 0
  33. 2025Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulationcited by 0
  34. 2025The Prompt is Mightier than the Examplecited by 0
  35. 2025CCQA: Generating Question from Solution Can Improve Inference-Time Reasoning in SLMscited by 0
  36. 2025BLAZER: Bootstrapping LLM-based Manipulation Agents with Zero-Shot Data Generationcited by 0
  37. 2025MMLongCite: A Benchmark for Evaluating Fidelity of Long-Context Vision-Language Modelscited by 0
  38. 2026Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptationcited by 0

50 most cited of 60 All 60 (JSON)