# Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Models

arXiv: https://arxiv.org/abs/2505.24187 · HTML: https://mlsanity.com/paper/2505.24187/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 15 of its 16 references that are in the map.
- First posted on arXiv 30 May 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 15 of 15)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025)
- [RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval](https://mlsanity.com/paper/2409.10516/) (2024)
- [What is Wrong with Perplexity for Long-context Language Modeling?](https://mlsanity.com/paper/2410.23771/) (2024)
- [TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection](https://mlsanity.com/paper/2411.02886/) (2024)
- [Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Models](https://mlsanity.com/paper/2309.02772/) (2023)
- [The Intrinsic Dimension of Prompts in Internal Representations of Large Language Models](https://mlsanity.com/paper/2501.10573/) (2025)
- [Token embeddings violate the manifold hypothesis](https://mlsanity.com/paper/2504.01002/) (2025)
- [Anchor-based Large Language Models](https://mlsanity.com/paper/2402.07616/) (2024)
- [Unraveling the Localized Latents: Learning Stratified Manifold Structures in LLM Embedding Space with Sparse Mixture-of-Experts](https://mlsanity.com/paper/2502.13577/) (2025)
- [Think, Prune, Train, Improve: Scaling Reasoning without Scaling Models](https://mlsanity.com/paper/2504.18116/) (2025)

## Built on by (most cited first, 12 of 12)

- [Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning](https://mlsanity.com/paper/2506.02867/) (2025)
- [ReMiT: RL-Guided Mid-Training for Iterative LLM Evolution](https://mlsanity.com/paper/2602.03075/) (2026)
- [Limited Reasoning Space: The cage of long-horizon reasoning in LLMs](https://mlsanity.com/paper/2602.19281/) (2026)
- [Uncertainty Propagation in LLM-Based Systems](https://mlsanity.com/paper/2604.23505/) (2026)
- [Mechanism-Based Intelligence (MBI): Differentiable Incentives for Rational Coordination and Guaranteed Alignment in Multi-Agent Systems](https://mlsanity.com/paper/2512.20688/) (2025)
- [LLM Flow Processes for Text-Conditioned Regression](https://mlsanity.com/paper/2601.06147/) (2026)
- [ReFlect: An Effective Harness System for Complex Long-Horizon LLM Reasoning](https://mlsanity.com/paper/2605.05737/) (2026)
- [DeepArrhythmia: Segment-Contextualized ECG Arrhythmia Classification via Selective Evidence Acquisition](https://mlsanity.com/paper/2605.16441/) (2026)
- [Peak-Detector: Explainable Peak Detection via Instruction-Tuned Large Language Models in Physiological Sign](https://mlsanity.com/paper/2605.16452/) (2026)
- [The Architecture of Errors: From Universal Impossibility to Patch-Local LLM Reliability](https://mlsanity.com/paper/2605.30628/) (2026)
- [Human-in-the-Loop Control of Objective Drift in LLM-Assisted Computer Science Education](https://mlsanity.com/paper/2604.00281/) (2026)
- [Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game](https://mlsanity.com/paper/2605.00677/) (2026)

## Abstract (arXiv)

The prevailing assumption of an exponential decay in large language model (LLM) reliability with sequence length, predicated on independent per-token error probabilities, posits an inherent limitation for long autoregressive outputs. Our research fundamentally challenges this view by synthesizing emerging evidence that LLM errors are not uniformly distributed but are concentrated at sparse "key tokens" (5-10% of total tokens) representing critical decision junctions. By distinguishing these high-impact tokens from the increasingly predictable majority, we introduce a new reliability formula explaining the sustained coherence of modern LLMs over thousands of tokens. Converging research streams reveal that long-context performance primarily depends on accurately navigating a few crucial semantic decision points rather than on uniform token-level accuracy, enabling targeted strategies that significantly outperform brute-force approaches. We thus propose a framework for next-generation systems centered on selective preservation of semantically vital tokens, dynamic computational allocation at uncertain decision boundaries, multi-path exploration at ambiguities, and architectures aligned with natural semantic domains. This marks a fundamental shift from raw scaling to strategic reasoning, promising breakthrough performance without proportionate computational scaling and offering a more nuanced understanding that supersedes the exponential decay hypothesis, thereby opening pathways toward substantially more powerful and efficient language systems.