# Shannon Entropy Rate of Hidden Markov Processes

arXiv: https://arxiv.org/abs/2008.12886 · HTML: https://mlsanity.com/paper/2008.12886/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2020.

## Built on by (most cited first, 8 of 8)

- [Transformers represent belief state geometry in their residual stream](https://mlsanity.com/paper/2405.15943/) (2024)
- [Topology, Convergence, and Reconstruction of Predictive States](https://mlsanity.com/paper/2109.09203/) (2021)
- [Next-token pretraining implies in-context learning](https://mlsanity.com/paper/2505.18373/) (2025)
- [From monoliths to modules: Decomposing transducers for efficient world modelling](https://mlsanity.com/paper/2512.02193/) (2025)
- [AI in a vat: Fundamental limits of efficient world modelling for agent sandboxing and interpretability](https://mlsanity.com/paper/2504.04608/) (2025)
- [Complexity-calibrated Benchmarks for Machine Learning Reveal When Next-Generation Reservoir Computer Predictions Succeed and Mislead](https://mlsanity.com/paper/2303.14553/) (2023)
- [Ultimate limit on learning non-Markovian behavior: Fisher information rate and excess information](https://mlsanity.com/paper/2310.03968/) (2023)
- [Inferring Kernel ε-Machines: Discovering Structure in Complex Systems](https://mlsanity.com/paper/2410.01076/) (2024)

## Abstract (arXiv)

Hidden Markov chains are widely applied statistical models of stochastic processes, from fundamental physics and chemistry to finance, health, and artificial intelligence. The hidden Markov processes they generate are notoriously complicated, however, even if the chain is finite state: no finite expression for their Shannon entropy rate exists, as the set of their predictive features is generically infinite. As such, to date one cannot make general statements about how random they are nor how structured. Here, we address the first part of this challenge by showing how to efficiently and accurately calculate their entropy rates. We also show how this method gives the minimal set of infinite predictive features. A sequel addresses the challenge's second part on structure.