# Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly

arXiv: https://arxiv.org/abs/2408.01866 · HTML: https://mlsanity.com/paper/2408.01866/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 33 references that are in the map.
- First posted on arXiv 3 Aug 2024.

## Builds on (most cited first, 6 of 6)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [TLDR: Extreme Summarization of Scientific Documents](https://mlsanity.com/paper/2004.15011/) (2020)
- [Unlimiformer: Long-Range Transformers with Unlimited Length Input](https://mlsanity.com/paper/2305.01625/) (2023)
- [LLM In-Context Recall is Prompt Dependent](https://mlsanity.com/paper/2404.08865/) (2024)

## Built on by (most cited first, 24 of 24)

- [Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management](https://mlsanity.com/paper/2510.06727/) (2025)
- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [According to Me: Long-Term Personalized Referential Memory QA](https://mlsanity.com/paper/2603.01990/) (2026)
- [Stop Wasting Your Tokens: Towards Efficient Runtime Multi-Agent Systems](https://mlsanity.com/paper/2510.26585/) (2025)
- [DAPE V2: Process Attention Score as Feature Map for Length Extrapolation](https://mlsanity.com/paper/2410.04798/) (2024)
- [MDCure: A Scalable Pipeline for Multi-Document Instruction-Following](https://mlsanity.com/paper/2410.23463/) (2024)
- [Exploring LLM Reasoning Through Controlled Prompt Variations](https://mlsanity.com/paper/2504.02111/) (2025)
- [TAMA: A Human-AI Collaborative Thematic Analysis Framework Using Multi-Agent LLMs for Clinical Interviews](https://mlsanity.com/paper/2503.20666/) (2025)
- [Retrieval-Augmented Test Generation: How Far Are We?](https://mlsanity.com/paper/2409.12682/) (2024)
- [CG-TTRL: Context-Guided Test-Time Reinforcement Learning for On-Device Large Language Models](https://mlsanity.com/paper/2511.06430/) (2025)
- [MASFIN: A Multi-Agent System for Decomposed Financial Reasoning and Forecasting](https://mlsanity.com/paper/2512.21878/) (2025)
- [AI-assisted Protocol Information Extraction For Improved Accuracy and Efficiency in Clinical Trial Workflows](https://mlsanity.com/paper/2602.00052/) (2026)
- [Evaluating adaptive and generative AI-based feedback and recommendations in a knowledge-graph-integrated programming learning system](https://mlsanity.com/paper/2603.24940/) (2026)
- [Cross-Lingual Attention Distillation with Personality-Informed Generative Augmentation for Multilingual Personality Recognition](https://mlsanity.com/paper/2604.08851/) (2026)
- [NTILC: Neural Tool Invocation via Learned Compression](https://mlsanity.com/paper/2606.06566/) (2026)
- [COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learning](https://mlsanity.com/paper/2608.30109/) (2026)
- [From Words to Workflows: Automating Business Processes](https://mlsanity.com/paper/2412.03446/) (2024)
- [MSRS: Evaluating Multi-Source Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.20867/) (2025)
- [GPT-5 vs Other LLMs in Long Short-Context Performance](https://mlsanity.com/paper/2602.14188/) (2026)
- [Can LLMs Reason Like Automated Theorem Provers for Rust Verification? VCoT-Bench: Evaluating via Verification Chain of Thought](https://mlsanity.com/paper/2603.18334/) (2026)
- [Fine-Tuning Small Language Models for Solution-Oriented Windows Event Log Analysis](https://mlsanity.com/paper/2605.06330/) (2026)
- [Enhancing Software Engineering Through Closed-Loop Memory Optimization](https://mlsanity.com/paper/2606.05646/) (2026)
- [Impacts of Histories and Models on LLM Grading: A Study in Advanced Software Engineering Courses](https://mlsanity.com/paper/2606.08400/) (2026)
- [LLMs for Academic Workflows: An Evaluation of Literature Reviews Generated with Short and Long Context Windows of LLMs](https://mlsanity.com/paper/2608.26145/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities in comprehending and analyzing lengthy sequential inputs, owing to their extensive context windows that allow processing millions of tokens in a single forward pass. However, this paper uncovers a surprising limitation: LLMs fall short when handling long input sequences. We investigate this issue using three datasets and two tasks (sentiment analysis and news categorization) across various LLMs, including Claude 3, Gemini Pro, GPT 3.5 Turbo, Llama 3 Instruct, and Mistral Instruct models. To address this limitation, we propose and evaluate ad-hoc solutions that substantially enhance LLMs' performance on long input sequences by up to 50%, while reducing API cost and latency by up to 93% and 50%, respectively.