# From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence

arXiv: https://arxiv.org/abs/2601.03220 · HTML: https://mlsanity.com/paper/2601.03220/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jan 2026.

## Built on by (most cited first, 30 of 41)

- [Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain](https://mlsanity.com/paper/2603.02218/) (2026)
- [Training Language Models via Neural Cellular Automata](https://mlsanity.com/paper/2603.10055/) (2026)
- [Shaping capabilities with token-level data filtering](https://mlsanity.com/paper/2601.21571/) (2026)
- [Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2601.19280/) (2026)
- [Data-efficient pre-training by scaling synthetic megadocs](https://mlsanity.com/paper/2603.18534/) (2026)
- [Human-like autonomy emerges from self-play and a pinch of human data](https://mlsanity.com/paper/2606.19370/) (2026)
- [Interestingness as an Inductive Heuristic for Future Compression Progress](https://mlsanity.com/paper/2605.14831/) (2026)
- [A Compositional Framework for Open-ended Intelligence](https://mlsanity.com/paper/2606.15386/) (2026)
- [Are We Ready for AI-Driven Discovery? AI Verification Before the Next Fundamental Physics Breakthrough](https://mlsanity.com/paper/2607.10039/) (2026)
- [Intelligence from Learnable Novelty](https://mlsanity.com/paper/2607.18433/) (2026)
- [Why Agentic Theorem Prover Works: A Statistical Provability Theory of Mathematical Reasoning Models](https://mlsanity.com/paper/2602.10538/) (2026)
- [On the Rate-Distortion-Complexity Tradeoff for Semantic Communication](https://mlsanity.com/paper/2602.14481/) (2026)
- [Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution](https://mlsanity.com/paper/2604.03472/) (2026)
- [SANEmerg: An Emergent Communication Framework for Semantic-aware Agentic AI Networking](https://mlsanity.com/paper/2605.05861/) (2026)
- [From Mechanistic to Compositional Interpretability](https://mlsanity.com/paper/2605.08934/) (2026)
- [Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data](https://mlsanity.com/paper/2607.11883/) (2026)
- [Self-Play Pretraining with Zero Data](https://mlsanity.com/paper/2609.30063/) (2026)
- [Thermodynamic Limits of Physical Intelligence](https://mlsanity.com/paper/2602.05463/) (2026)
- [X-RAY: Mapping LLM Reasoning Capability via Formalized and Calibrated Probes](https://mlsanity.com/paper/2603.05290/) (2026)
- [Why Does It Look There? Structured Explanations for Image Classification](https://mlsanity.com/paper/2603.10234/) (2026)
- [Predicting LLM Compression Degradation from Spectral Statistics](https://mlsanity.com/paper/2604.18085/) (2026)
- [Generalization Bounds of Emergent Communications for Agentic AI Networking](https://mlsanity.com/paper/2605.08613/) (2026)
- [A Controlled Counterexample to Strong Proxy-Based Explanations of OOD Performance: in a Fixed Pretraining-and-Probing Setup](https://mlsanity.com/paper/2605.11554/) (2026)
- [Seeking the Unfamiliar but Memorable: Conceptual Creativity as Meta-Learning](https://mlsanity.com/paper/2605.16477/) (2026)
- [On Distributional Reinforcement Learning in Chaotic Dynamical Systems](https://mlsanity.com/paper/2605.30160/) (2026)
- [Inductive Generalization for Robotic Manipulation](https://mlsanity.com/paper/2606.20999/) (2026)
- [Evaluating self-supervised echocardiographic representations across downstream extraction strategies for left-ventricular segmentation and ejection fraction estimation](https://mlsanity.com/paper/2606.22943/) (2026)
- [TSD: A Physics-Inspired Trajectory Saliency Detector for Efficient Imitation Learning](https://mlsanity.com/paper/2606.23371/) (2026)
- [SIEVE: Structure-Aware Data Selection for Imitation Learning with VLA Models](https://mlsanity.com/paper/2607.06442/) (2026)
- [Mirror Horizon: Viable Path Entropy as a Measure of Bounded Reflection](https://mlsanity.com/paper/2607.11937/) (2026)

## Abstract (arXiv)

Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated without considering a downstream task? On these questions, Shannon information and Kolmogorov complexity come up nearly empty-handed, in part because they assume observers with unlimited computational capacity and do not target the useful information content. In this work, we identify and exemplify three seeming paradoxes in information theory: (1) information cannot be increased by deterministic transformations; (2) information is independent of the order of data; (3) likelihood modeling is merely distribution matching. To shed light on the tension between these results and modern practice, and to quantify the value of data, we introduce epiplexity, a formalization of information capturing what computationally bounded observers can learn from data. Epiplexity captures the structural content in data while excluding time-bounded entropy, the random unpredictable content exemplified by pseudorandom number generators and chaotic dynamical systems. With these concepts, we demonstrate how information can be created with computation, how it depends on the ordering of the data, and how likelihood modeling can produce more complex programs than present in the data generating process itself. We also present practical procedures to estimate epiplexity which we show capture differences across data sources, track with downstream performance, and highlight dataset interventions that improve out-of-distribution generalization. In contrast to principles of model selection, epiplexity provides a theoretical foundation for data selection, guiding how to select, generate, or transform data for learning systems.