# On the Fundamental Limits of LLMs at Scale

arXiv: https://arxiv.org/abs/2511.12869 · HTML: https://mlsanity.com/paper/2511.12869/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- First posted on arXiv 17 Nov 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Built on by (most cited first, 18 of 18)

- [Large Language Model Agents Are Not Always Faithful Self-Evolvers](https://mlsanity.com/paper/2601.22436/) (2026)
- [Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testing](https://mlsanity.com/paper/2602.00906/) (2026)
- [Emergence of Fragility in LLM-based Social Networks: the Case of Moltbook](https://mlsanity.com/paper/2603.23279/) (2026)
- [Persona-Based Simulation of Human Opinion at Population Scale](https://mlsanity.com/paper/2603.27056/) (2026)
- [An Alternative Trajectory for Generative AI](https://mlsanity.com/paper/2603.14147/) (2026)
- [From Unstructured Recall to Schema-Grounded Memory: Reliable AI Memory via Iterative, Schema-Aware Extraction](https://mlsanity.com/paper/2604.27906/) (2026)
- [Shaping Schema via Language Representation as the Next Frontier for LLM Intelligence Expanding](https://mlsanity.com/paper/2605.09271/) (2026)
- [Liquid Interfaces: A Dynamic Ontology for the Interoperability of Autonomous Systems](https://mlsanity.com/paper/2601.21993/) (2026)
- [Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition](https://mlsanity.com/paper/2603.16463/) (2026)
- [How Psychological Learning Paradigms Shaped and Constrained Artificial Intelligence](https://mlsanity.com/paper/2603.18203/) (2026)
- [Strengthening Human-Centric Chain-of-Thought Reasoning Integrity in LLMs via a Structured Prompt Framework](https://mlsanity.com/paper/2604.04852/) (2026)
- [Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgments](https://mlsanity.com/paper/2605.03971/) (2026)
- [REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak](https://mlsanity.com/paper/2605.20654/) (2026)
- [The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems](https://mlsanity.com/paper/2605.23024/) (2026)
- [Limits of Reliability and Scaling in Language Models](https://mlsanity.com/paper/2607.14112/) (2026)
- [Eliciting Intrinsic Hallucinations in LLMs via Semantically Equivalent Adversarial Attacks](https://mlsanity.com/paper/2608.04286/) (2026)
- [Social Chain of Thought: A Multi-Agent Architecture Grounded in Medical Differential Diagnosis Methodology](https://mlsanity.com/paper/2608.11420/) (2026)
- [Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oracle](https://mlsanity.com/paper/2608.30581/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have benefited enormously from scaling, yet these gains are bounded by five fundamental limitations: (1) hallucination, (2) context compression, (3) reasoning degradation, (4) retrieval fragility, and (5) multimodal misalignment. While existing surveys describe these phenomena empirically, they lack a rigorous theoretical synthesis connecting them to the foundational limits of computation, information, and learning. This work closes that gap by presenting a unified, proof-informed framework that formalizes the innate theoretical ceilings of LLM scaling. First, computability and uncomputability imply an irreducible residue of error: for any computably enumerable model family, diagonalization guarantees inputs on which some model must fail, and undecidable queries (e.g., halting-style tasks) induce infinite failure sets for all computable predictors. Second, information-theoretic and statistical constraints bound attainable accuracy even on decidable tasks, finite description length enforces compression error, and long-tail factual knowledge requires prohibitive sample complexity. Third, geometric and computational effects compress long contexts far below their nominal size due to positional under-training, encoding attenuation, and softmax crowding. We further show how likelihood-based training favors pattern completion over inference, how retrieval under token limits suffers from semantic drift and coupling noise, and how multimodal scaling inherits shallow cross-modal alignment. Across sections, we pair theorems and empirical evidence to outline where scaling helps, where it saturates, and where it cannot progress, providing both theoretical foundations and practical mitigation paths like bounded-oracle retrieval, positional curricula, and sparse or hierarchical attention.