# Why the Valuable Capabilities of LLMs Are Precisely the Unexplainable Ones

arXiv: https://arxiv.org/abs/2603.15238 · HTML: https://mlsanity.com/paper/2603.15238/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 22 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 5 of 5)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [The Limits of AI Explainability: An Algorithmic Information Theory Approach](https://mlsanity.com/paper/2504.20676/) (2025)
- [G"odel Incompleteness Theorem for PAC Learnable Theory from the view of complexity measurement](https://mlsanity.com/paper/2408.10211/) (2024)

## Built on by (most cited first, 1 of 1)

- [Via Negativa for AI Alignment: Why Negative Constraints Are Structurally Superior to Positive Preferences](https://mlsanity.com/paper/2603.16417/) (2026)

## Abstract (arXiv)

This paper proposes and argues for a counterintuitive thesis: the truly valuable capabilities of large language models (LLMs) reside precisely in the part that cannot be fully captured by human-readable discrete rules. The core argument is a proof by contradiction via expert system equivalence: if the full capabilities of an LLM could be described by a complete set of human-readable rules, then that rule set would be functionally equivalent to an expert system; but expert systems have been historically and empirically demonstrated to be strictly weaker than LLMs; therefore, a contradiction arises -- the capabilities of LLMs that exceed those of expert systems are exactly the capabilities that cannot be rule-encoded. This thesis is further supported by the Chinese philosophical concept of Wu (sudden insight through practice), the historical failure of expert systems, and a structural mismatch between human cognitive tools and complex systems. The paper discusses implications for interpretability research, AI safety, and scientific epistemology.