mlsanityData to 7 Oct 2026

Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly

arXiv 2408.01866 ↗First posted 3 Aug 2024cs.CL cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities in comprehending and analyzing lengthy sequential inputs, owing to their extensive context windows that allow processing millions of tokens in a single forward pass. However, this paper uncovers a surprising limitation: LLMs fall short when handling long input sequences. We investigate this issue using three datasets and two tasks (sentiment analysis and news categorization) across various LLMs, including Claude 3, Gemini Pro, GPT 3.5 Turbo, Llama 3 Instruct, and Mistral Instruct models. To address this limitation, we propose and evaluate ad-hoc solutions that substantially enhance LLMs' performance on long input sequences by up to 50%, while reducing API cost and latency by up to 93% and 50%, respectively.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 6 of its 33 references that are in the map. First posted on arXiv 3 Aug 2024.

Builds on

6 of 33 references.

References · 6

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  4. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
  5. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  6. 2024LLM In-Context Recall is Prompt Dependentcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 11.

23 papers.

Sort:

By area:

  1. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Managementcited by 51
  2. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  3. 2026According to Me: Long-Term Personalized Referential Memory QAcited by 12
  4. 2025Stop Wasting Your Tokens: Towards Efficient Runtime Multi-Agent Systemscited by 11
  5. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  6. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  7. 2025Exploring LLM Reasoning Through Controlled Prompt Variationscited by 10
  8. 2025TAMA: A Human-AI Collaborative Thematic Analysis Framework Using Multi-Agent LLMs for Clinical Interviewscited by 9
  9. 2024Retrieval-Augmented Test Generation: How Far Are We?cited by 7
  10. 2025CG-TTRL: Context-Guided Test-Time Reinforcement Learning for On-Device Large Language Modelscited by 1
  11. 2025MASFIN: A Multi-Agent System for Decomposed Financial Reasoning and Forecastingcited by 1
  12. 2026Evaluating adaptive and generative AI-based feedback and recommendations in a knowledge-graph-integrated programming learning systemcited by 1
11 more
  1. 2026NTILC: Neural Tool Invocation via Learned Compressioncited by 1
  2. 2026COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learningcited by 1
  3. 2024From Words to Workflows: Automating Business Processescited by 0
  4. 2025MSRS: Evaluating Multi-Source Retrieval-Augmented Generationcited by 0
  5. 2026AI-assisted Protocol Information Extraction For Improved Accuracy and Efficiency in Clinical Trial Workflowscited by 0
  6. 2026GPT-5 vs Other LLMs in Long Short-Context Performancecited by 0
  7. 2026Can LLMs Reason Like Automated Theorem Provers for Rust Verification? VCoT-Bench: Evaluating via Verification Chain of Thoughtcited by 0
  8. 2026Fine-Tuning Small Language Models for Solution-Oriented Windows Event Log Analysiscited by 0
  9. 2026Enhancing Software Engineering Through Closed-Loop Memory Optimizationcited by 0
  10. 2026Impacts of Histories and Models on LLM Grading: A Study in Advanced Software Engineering Coursescited by 0
  11. 2026LLMs for Academic Workflows: An Evaluation of Literature Reviews Generated with Short and Long Context Windows of LLMscited by 0