# The Information-Theoretic Imperative: Compression and the Epistemic Foundations of Intelligence

arXiv: https://arxiv.org/abs/2510.25883 · HTML: https://mlsanity.com/paper/2510.25883/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 78 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 16 of 16)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [Active inference on discrete state-spaces: a synthesis](https://mlsanity.com/paper/2001.07203/) (2020)
- [Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning](https://mlsanity.com/paper/1703.01732/) (2017)
- [The free energy principle for action and perception: A mathematical review](https://mlsanity.com/paper/1705.09156/) (2017)
- [The deterministic information bottleneck](https://mlsanity.com/paper/1604.00268/) (2016)
- [Machine learning and information theory concepts towards an AI Mathematician](https://mlsanity.com/paper/2403.04571/) (2024)
- [Evaluating Large Language Models for Generalization and Robustness via Data Compression](https://mlsanity.com/paper/2402.00861/) (2024)
- [Curiosity as filling, compressing, and reconfiguring knowledge networks](https://mlsanity.com/paper/2204.01182/) (2022)

## Built on by (most cited first, 1 of 1)

- [Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI](https://mlsanity.com/paper/2603.25033/) (2026)

## Abstract (arXiv)

Why do brains and deep networks converge on similar representations? Task-optimized artificial neural networks quantitatively predict primate ventral stream responses despite radically different substrates and optimization dynamics. This convergence demands explanation beyond shared natural image statistics or task structure alone. The Compression Efficiency Principle (CEP) specifies the selection mechanism: representations exploiting unstable correlations pay a growing "exception tax" (approximately linear excess codelength under shortcut-flipping shifts), while representations encoding shift-stable invariants amortize this cost. When environments provide intervention-rich shifts and exhibit approximately modular causal structure, these invariants align with causal mechanisms. The framework offers a unified lens on three biological signatures -- steep metabolic constraints on neural signaling, high coding efficiency in early sensory pathways, and hierarchical tolerance in the ventral stream -- and connects them to parallel phenomena in deep learning: scaling frontiers, shortcut failures under distribution shift, and the role of augmentation in enforcing invariances. Distinctive predictions follow: a crossover threshold beyond which invariant representations dominate, and systematic coupling between compression efficiency and out-of-distribution robustness -- testable across substrates. Predicted divergences (sparse biological signaling versus dense overparameterization) arise from different resource constraints on a shared trade-off topology. The convergence is not a coincidence. It is evidence for a substrate-independent basin shaped by predictive compression under shift.