# Nearly Maximally Predictive Features and Their Dimensions

arXiv: https://arxiv.org/abs/1702.08565 · HTML: https://mlsanity.com/paper/1702.08565/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2017.

## Built on by (most cited first, 9 of 9)

- [Transformers represent belief state geometry in their residual stream](https://mlsanity.com/paper/2405.15943/) (2024)
- [Generative learning for nonlinear dynamics](https://mlsanity.com/paper/2311.04128/) (2023)
- [Constrained belief updates explain geometric structures in transformer representations](https://mlsanity.com/paper/2502.01954/) (2025)
- [Transformers learn factored representations](https://mlsanity.com/paper/2602.02385/) (2026)
- [Neural networks leverage nominally quantum and post-quantum representations](https://mlsanity.com/paper/2507.07432/) (2025)
- [From monoliths to modules: Decomposing transducers for efficient world modelling](https://mlsanity.com/paper/2512.02193/) (2025)
- [Complexity-calibrated Benchmarks for Machine Learning Reveal When Next-Generation Reservoir Computer Predictions Succeed and Mislead](https://mlsanity.com/paper/2303.14553/) (2023)
- [Structure and Scale in Simplicial Sequence Modelling](https://mlsanity.com/paper/2606.01302/) (2026)
- [An Irreducible Quantum Advantage in Aligning World Models with Reality](https://mlsanity.com/paper/2608.19779/) (2026)

## Abstract (arXiv)

Scientific explanation often requires inferring maximally predictive features from a given data set. Unfortunately, the collection of minimal maximally predictive features for most stochastic processes is uncountably infinite. In such cases, one compromises and instead seeks nearly maximally predictive features. Here, we derive upper-bounds on the rates at which the number and the coding cost of nearly maximally predictive features scales with desired predictive power. The rates are determined by the fractal dimensions of a process' mixed-state distribution. These results, in turn, show how widely-used finite-order Markov models can fail as predictors and that mixed-state predictive features offer a substantial improvement.