mlsanityData to 7 Oct 2026

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models

arXiv 2402.14848 ↗First posted 19 Feb 2024cs.CL cs.AIcited by 150

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

This paper explores the impact of extending input lengths on the capabilities of Large Language Models (LLMs). Despite LLMs advancements in recent times, their performance consistency across different input lengths is not well understood. We investigate this aspect by introducing a novel QA reasoning framework, specifically designed to assess the impact of input length. We isolate the effect of input length using multiple versions of the same sample, each being extended with padding of different lengths, types and locations. Our findings show a notable degradation in LLMs' reasoning performance at much shorter input lengths than their technical maximum. We show that the degradation trend appears in every version of our dataset, although at different intensities. Additionally, our study reveals that the traditional metric of next word prediction correlates negatively with performance of LLMs' on our reasoning dataset. We analyse our results and identify failure modes that can serve as useful guides for future research, potentially informing strategies to address the limitations observed in LLMs.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 26 of its 41 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

26 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Language models show human-like content effects on reasoning tasks14 Jul

Other references · 25

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  8. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  9. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  10. 2020Transformers as Soft Reasoners over Languagecited by 265
  11. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  12. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
13 more
  1. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  2. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  3. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  4. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  5. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  6. 2019Compositional Questions Do Not Necessitate Multi-hop Reasoningcited by 87
  7. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  8. 2022Training Trajectories of Language Models Across Scalescited by 80
  9. 2019Understanding Dataset Design Choices for Multi-hop Reasoningcited by 50
  10. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  11. 2021End-to-End Segmentation-based News Summarizationcited by 16
  12. 2023Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggerscited by 10
  13. 2018Compositional Language Understanding with Text-based Relational Reasoningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 45; 2025: 61; 2026: 44.

150 papers.

Sort:

By area:

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  3. 2025Phi-4-reasoning Technical Reportcited by 146
  4. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  5. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  6. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  7. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 109
  8. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 85
  9. 2024Language Repository for Long Video Understandingcited by 55
  10. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
  11. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  12. 2024Disentangling Memory and Reasoning Ability in Large Language Modelscited by 43
38 more
  1. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longercited by 39
  2. 2024OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scalecited by 37
  3. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 33
  4. 2024LaMMA-P: Generalizable Multi-Agent Long-Horizon Task Allocation and Planning with LM-Driven PDDL Plannercited by 31
  5. 2024Not All LLM Reasoners Are Created Equalcited by 29
  6. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  7. 2025Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Contextcited by 24
  8. 2024When Can Transformers Count to n?cited by 22
  9. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Examplecited by 22
  10. 2024Context-Aware Membership Inference Attacks against Pre-trained Large Language Modelscited by 20
  11. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  12. 2024Markov Chain of Thought for Efficient Mathematical Reasoningcited by 19
  13. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  14. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  15. 2024Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?cited by 17
  16. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?cited by 16
  17. 2025Whitened CLIP as a Likelihood Surrogate of Images and Captionscited by 16
  18. 2026Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialoguescited by 16
  19. 2025Intrinsic Entropy of Context Length Scaling in LLMscited by 14
  20. 2025SiLVR: A Simple Language-based Video Reasoning Frameworkcited by 14
  21. 2024MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMscited by 13
  22. 2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QAcited by 13
  23. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
  24. 2024Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Datacited by 11
  25. 2025Episodic memory in AI agents poses risks that should be studied and mitigatedcited by 11
  26. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 11
  27. 2024CT-AGRG: Automated Abnormality-Guided Report Generation from 3D Chest CT Volumescited by 10
  28. 2024Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairmentscited by 10
  29. 2025United Minds or Isolated Agents? Exploring Coordination of LLMs under Cognitive Load Theorycited by 10
  30. 2024ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Modelscited by 9
  31. 2024Statically Contextualizing Large Language Models with Typed Holescited by 9
  32. 2024Assessing Episodic Memory in LLMs with Sequence Order Recall Taskscited by 9
  33. 2025Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMscited by 9
  34. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 9
  35. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  36. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  37. 2025Positional Biases Shift as Inputs Approach Context Window Limitscited by 9
  38. 2024Toward Conversational Agents with Context and Time Sensitive Long-term Memorycited by 8

50 most cited of 150 All 150 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint
  2. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
  3. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  4. 2025The Art of Scaling Test-Time Compute for Large Language Models
  5. 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
  6. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs
  7. 2025Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Models
  8. 2026Reasoning Shift: How Context Silently Shortens LLM Reasoning