# Intelligence Degradation in Long-Context LLMs: Critical Threshold Determination via Natural Length Distribution Analysis

arXiv: https://arxiv.org/abs/2601.15300 · HTML: https://mlsanity.com/paper/2601.15300/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Builds on (most cited first, 11 of 11)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models](https://mlsanity.com/paper/2309.12307/) (2023)
- [Context Length Alone Hurts LLM Performance Despite Perfect Retrieval](https://mlsanity.com/paper/2510.05381/) (2025)
- [Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training](https://mlsanity.com/paper/2311.09198/) (2023)
- [LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges?](https://mlsanity.com/paper/2510.22548/) (2025)
- [FltLM: An Intergrated Long-Context Large Language Model for Effective Context Filtering and Understanding](https://mlsanity.com/paper/2410.06886/) (2024)

## Built on by (most cited first, 4 of 4)

- [Rethinking Code Complexity Through the Lens of Large Language Models](https://mlsanity.com/paper/2602.07882/) (2026)
- [WMF-AM: Probing LLM Working Memory via Depth-Parameterized Cumulative State Tracking](https://mlsanity.com/paper/2603.27343/) (2026)
- [Context Cartography: Toward Structured Governance of Contextual Space in Large Language Model Systems](https://mlsanity.com/paper/2603.20578/) (2026)
- [NTILC: Neural Tool Invocation via Learned Compression](https://mlsanity.com/paper/2606.06566/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) exhibit catastrophic performance degradation when processing contexts approaching certain critical thresholds, even when information remains relevant. This intelligence degradation-defined as over 30% drop in task performance-severely limits long-context applications. This degradation shows a common pattern: models maintain strong performance up to a critical threshold, then collapse catastrophically. We term this shallow long-context adaptation-models adapt for short to medium contexts but fail beyond critical thresholds. This paper presents three contributions: (1) Natural Length Distribution Analysis: We use each sample's natural token length without truncation or padding, providing stronger causal evidence that degradation results from context length itself. (2) Critical Threshold Determination: Through experiments on a mixed dataset (1,000 samples covering 5%-95% of context length), we identify the critical threshold for Qwen2.5-7B at 40-50% of maximum context length, where F1 scores drop from 0.55-0.56 to 0.3 (45.5% degradation), using five-method cross-validation. (3) Unified Framework: We consolidate shallow adaptation, explaining degradation patterns and providing a foundation for mitigation strategies. This work provides the first systematic characterization of intelligence degradation in open-source Qwen models, offering practical guidance for deploying LLMs in long-context scenarios.