# Truly Bayesian Entropy Estimation

arXiv: https://arxiv.org/abs/2212.06705 · HTML: https://mlsanity.com/paper/2212.06705/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2022.

## Built on by (most cited first, 3 of 3)

- [The Bayesian Context Trees State Space Model for time series modelling and forecasting](https://mlsanity.com/paper/2308.00913/) (2023)
- [Soft Bayesian Context Tree Models for Real-Valued Time Series](https://mlsanity.com/paper/2601.11079/) (2026)
- [Variable Splitting Binary Tree Models Based on Bayesian Context Tree Models for Time Series Segmentation](https://mlsanity.com/paper/2601.16112/) (2026)

## Abstract (arXiv)

Estimating the entropy rate of discrete time series is a challenging problem with important applications in numerous areas including neuroscience, genomics, image processing and natural language processing. A number of approaches have been developed for this task, typically based either on universal data compression algorithms, or on statistical estimators of the underlying process distribution. In this work, we propose a fully-Bayesian approach for entropy estimation. Building on the recently introduced Bayesian Context Trees (BCT) framework for modelling discrete time series as variable-memory Markov chains, we show that it is possible to sample directly from the induced posterior on the entropy rate. This can be used to estimate the entire posterior distribution, providing much richer information than point estimates. We develop theoretical results for the posterior distribution of the entropy rate, including proofs of consistency and asymptotic normality. The practical utility of the method is illustrated on both simulated and real-world data, where it is found to outperform state-of-the-art alternatives.