mlsanityData to 7 Oct 2026

Context Discipline and Performance Correlation: Analyzing LLM Performance and Quality Degradation Under Varying Context Lengths

arXiv 2601.11564 ↗First posted 25 Dec 2025cs.CL cs.AIcited by 2

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The scaling trend in Large Language Models (LLMs) has prioritized increasing the maximum context window to facilitate complex, long-form reasoning and document analysis. However, managing this expanded context introduces severe computational overhead. This paper investigates the critical trade-off between system performance and model quality when dense transformer architectures--specifically Llama-3.1-70B and Qwen1.5-14B--are exposed to large volumes of irrelevant and distracting context. The research identifies a non-linear performance degradation tied to the growth of the Key-Value (KV) cache. Furthermore, an extended analysis of the Mixture-of-Experts (MoE) architecture reveals unique behavioral anomalies at varying context scales, suggesting that architectural benefits may be masked by infrastructure bottlenecks at high token volumes.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 6 of its 7 references that are in the map. First posted on arXiv 25 Dec 2025.

Builds on

6 of 7 references.

References · 6

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2022Efficiently Scaling Transformer Inferencecited by 529
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  6. 2024On the Reliability of Large Language Models to Misinformed and Demographically-Informed Promptscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026World-Model Collapse as a Phase Transitioncited by 1
  2. 2026When Cloud Agents Meet Device Agents: Lessons from Hybrid Multi-Agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  2. 2025LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
  3. 2025On the Fundamental Limits of LLMs at Scale
  4. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
  5. 2025A Survey on Transformer Context Extension: Approaches and Evaluation
  6. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  7. 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell
  8. 2025EfficientLLM: Efficiency in Large Language Models