# On the Estimation of Information Measures of Continuous Distributions

arXiv: https://arxiv.org/abs/2002.02851 · HTML: https://mlsanity.com/paper/2002.02851/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2020.

## Built on by (most cited first, 6 of 6)

- [A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations](https://mlsanity.com/paper/2105.02685/) (2021)
- [A Differential Entropy Estimator for Training Neural Networks](https://mlsanity.com/paper/2202.06618/) (2022)
- [Learning Disentangled Textual Representations via Statistical Measures of Similarity](https://mlsanity.com/paper/2205.03589/) (2022)
- [PACMAN: PAC-style bounds accounting for the Mismatch between Accuracy and Negative log-loss](https://mlsanity.com/paper/2112.05547/) (2021)
- [A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification](https://mlsanity.com/paper/2310.13990/) (2023)
- [Statistical Deficiency for Task Inclusion Estimation](https://mlsanity.com/paper/2503.05491/) (2025)

## Abstract (arXiv)

The estimation of information measures of continuous distributions based on samples is a fundamental problem in statistics and machine learning. In this paper, we analyze estimates of differential entropy in K-dimensional Euclidean space, computed from a finite number of samples, when the probability density function belongs to a predetermined convex family P. First, estimating differential entropy to any accuracy is shown to be infeasible if the differential entropy of densities in P is unbounded, clearly showing the necessity of additional assumptions. Subsequently, we investigate sufficient conditions that enable confidence bounds for the estimation of differential entropy. In particular, we provide confidence bounds for simple histogram based estimation of differential entropy from a fixed number of samples, assuming that the probability density function is Lipschitz continuous with known Lipschitz constant and known, bounded support. Our focus is on differential entropy, but we provide examples that show that similar results hold for mutual information and relative entropy as well.