# Context Length Alone Hurts LLM Performance Despite Perfect Retrieval

arXiv: https://arxiv.org/abs/2510.05381 · HTML: https://mlsanity.com/paper/2510.05381/

## Key facts

- Cited by 127 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 57 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Builds on (most cited first, 30 of 42)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [Landmark Attention: Random-Access Infinite Context Length for Transformers](https://mlsanity.com/paper/2305.16300/) (2023)
- [Data Engineering for Scaling Language Models to 128K Context](https://mlsanity.com/paper/2402.10171/) (2024)
- [BooookScore: A systematic exploration of book-length summarization in the era of LLMs](https://mlsanity.com/paper/2310.00785/) (2023)
- [NoLiMa: Long-Context Evaluation Beyond Literal Matching](https://mlsanity.com/paper/2502.05167/) (2025)
- [Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG](https://mlsanity.com/paper/2410.05983/) (2024)
- [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://mlsanity.com/paper/2406.17419/) (2024)
- [LongAlign: A Recipe for Long Context Alignment of Large Language Models](https://mlsanity.com/paper/2401.18058/) (2024)
- [S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models](https://mlsanity.com/paper/2505.07686/) (2025)
- [One Thousand and One Pairs: A "novel" challenge for long-context language models](https://mlsanity.com/paper/2406.16264/) (2024)
- [Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding](https://mlsanity.com/paper/2403.04797/) (2024)
- [Efficient Long-Text Understanding with Short-Text Models](https://mlsanity.com/paper/2208.00748/) (2022)
- [Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?](https://mlsanity.com/paper/2502.12215/) (2025)
- [Why Does the Effective Context Length of LLMs Fall Short?](https://mlsanity.com/paper/2410.18745/) (2024)
- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [In Defense of RAG in the Era of Long-Context Language Models](https://mlsanity.com/paper/2409.01666/) (2024)
- [LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion](https://mlsanity.com/paper/2501.15089/) (2025)
- [RAR-b: Reasoning as Retrieval Benchmark](https://mlsanity.com/paper/2404.06347/) (2024)
- [RepoQA: Evaluating Long Context Code Understanding](https://mlsanity.com/paper/2406.06025/) (2024)
- [Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP](https://mlsanity.com/paper/2407.00402/) (2024)

## Built on by (most cited first, 30 of 127)

- [From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents](https://mlsanity.com/paper/2604.20006/) (2026)
- [SkillReducer: Optimizing LLM Agent Skills for Token Efficiency](https://mlsanity.com/paper/2603.29919/) (2026)
- [Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets](https://mlsanity.com/paper/2604.02460/) (2026)
- [GAM: Hierarchical Graph-based Agentic Memory for LLM Agents](https://mlsanity.com/paper/2604.12285/) (2026)
- [APEX-MEM: Agentic Semi-Structured Memory with Temporal Reasoning for Long-Term Conversational AI](https://mlsanity.com/paper/2604.14362/) (2026)
- [SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding Agents](https://mlsanity.com/paper/2606.28434/) (2026)
- [On Safety Risks in Experience-Driven Self-Evolving Agents](https://mlsanity.com/paper/2604.16968/) (2026)
- [δ-mem: Efficient Online Memory for Large Language Models](https://mlsanity.com/paper/2605.12357/) (2026)
- [Evaluating Memory Structure in LLM Agents](https://mlsanity.com/paper/2602.11243/) (2026)
- [Structurally Aligned Subtask-Level Memory for Software Engineering Agents](https://mlsanity.com/paper/2602.21611/) (2026)
- [Coding Agents are Effective Long-Context Processors](https://mlsanity.com/paper/2603.20432/) (2026)
- [SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?](https://mlsanity.com/paper/2605.26548/) (2026)
- [ContextPilot: Fast Long-Context Inference via Context Reuse](https://mlsanity.com/paper/2511.03475/) (2025)
- [Position: Modular Memory is the Key to Continual Learning Agents](https://mlsanity.com/paper/2603.01761/) (2026)
- [RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably](https://mlsanity.com/paper/2605.15514/) (2026)
- [Improving Coherence and Persistence in Agentic AI for System Optimization](https://mlsanity.com/paper/2603.21321/) (2026)
- [RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention](https://mlsanity.com/paper/2606.06256/) (2026)
- [Gated Differentiable Working Memory for Long-Context Language Modeling](https://mlsanity.com/paper/2601.12906/) (2026)
- [QUASAR: A Universal Autonomous System for Atomistic Simulation and a Benchmark of Its Capabilities](https://mlsanity.com/paper/2602.00185/) (2026)
- [EvoGuard: An Extensible Agentic RL-based Framework for Practical and Evolving AI-Generated Image Detection](https://mlsanity.com/paper/2603.17343/) (2026)
- [Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agents](https://mlsanity.com/paper/2605.08580/) (2026)
- [The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Management](https://mlsanity.com/paper/2605.23071/) (2026)
- [AgentSPEX: An Agent SPecification and EXecution Language](https://mlsanity.com/paper/2604.13346/) (2026)
- [Parallel Context Compaction for Long-Horizon LLM Agent Serving](https://mlsanity.com/paper/2605.23296/) (2026)
- [Intelligence Degradation in Long-Context LLMs: Critical Threshold Determination via Natural Length Distribution Analysis](https://mlsanity.com/paper/2601.15300/) (2026)
- [Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches](https://mlsanity.com/paper/2602.08561/) (2026)
- [Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agents](https://mlsanity.com/paper/2510.14453/) (2025)
- [Training-Free Active Learning Framework in Materials Science with Large Language Models](https://mlsanity.com/paper/2511.19730/) (2025)
- [A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challenges](https://mlsanity.com/paper/2602.05883/) (2026)
- [Agent Skill Framework: Perspectives on the Potential of Small to Medium Language Models in Industrial Environments](https://mlsanity.com/paper/2602.16653/) (2026)

## Abstract (arXiv)

Large language models (LLMs) often fail to scale their performance on long-context tasks performance in line with the context lengths they support. This gap is commonly attributed to retrieval failures -- the models' inability to identify relevant information in the long inputs. Accordingly, recent efforts often focus on evaluating and improving LLMs' retrieval performance: if retrieval is perfect, a model should, in principle, perform just as well on a long input as it does on a short one -- or should it? This paper presents findings that the answer to this question may be negative. Our systematic experiments across 5 open- and closed-source LLMs on math, question answering, and coding tasks reveal that, even when models can perfectly retrieve all relevant information, their performance still degrades substantially (13.9%--85%) as input length increases but remains well within the models' claimed lengths. This failure occurs even when the irrelevant tokens are replaced with minimally distracting whitespace, and, more surprisingly, when they are all masked and the models are forced to attend only to the relevant tokens. A similar performance drop is observed when all relevant evidence is placed immediately before the question. Our findings reveal a previously-unrealized limitation: the sheer length of the input alone can hurt LLM performance, independent of retrieval quality and without any distraction. They motivate our simple, model-agnostic mitigation strategy that transforms a long-context task into a short-context one by prompting the model to recite the retrieved evidence before attempting to solve the problem. On RULER, we observe a consistent improvement of GPT-4o up to 4% on an already strong baseline.