# Optimal rates of entropy estimation over Lipschitz balls

arXiv: https://arxiv.org/abs/1711.02141 · HTML: https://mlsanity.com/paper/1711.02141/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2017.

## Built on by (most cited first, 16 of 16)

- [Plugin Estimation of Smooth Optimal Transport Maps](https://mlsanity.com/paper/2107.12364/) (2021)
- [Information Theory with Kernel Methods](https://mlsanity.com/paper/2202.08545/) (2022)
- [The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models](https://mlsanity.com/paper/2204.10759/) (2022)
- [Neural Estimation of Statistical Divergences](https://mlsanity.com/paper/2110.03652/) (2021)
- [Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression](https://mlsanity.com/paper/2305.08013/) (2023)
- [Non-Asymptotic Performance Guarantees for Neural Estimation of mathsff-Divergences](https://mlsanity.com/paper/2103.06923/) (2021)
- [Efficient Bayesian network structure learning via local Markov boundary search](https://mlsanity.com/paper/2110.06082/) (2021)
- [Structure Learning for Directed Trees](https://mlsanity.com/paper/2108.08871/) (2021)
- [The Representation Jensen-Shannon Divergence](https://mlsanity.com/paper/2305.16446/) (2023)
- [Adversarial Information Bottleneck](https://mlsanity.com/paper/2103.00381/) (2021)
- [Reinforcement Learning with Exogenous States and Rewards](https://mlsanity.com/paper/2303.12957/) (2023)
- [When can Regression-Adjusted Control Variates Help? Rare Events, Sobolev Embedding and Minimax Optimality](https://mlsanity.com/paper/2305.16527/) (2023)
- [Efficient Alternating Minimization Solvers for Wyner Multi-View Unsupervised Learning](https://mlsanity.com/paper/2303.15866/) (2023)
- [Locally differentially private estimation of nonlinear functionals of discrete distributions](https://mlsanity.com/paper/2107.03940/) (2021)
- [Tree density estimation](https://mlsanity.com/paper/2111.11971/) (2021)
- [Copula Active Subspaces II: Error Decomposition, A Posteriori Estimation, and Sharpness of the Bounds](https://mlsanity.com/paper/2610.14498/) (2026)

## Abstract (arXiv)

We consider the problem of minimax estimation of the entropy of a density over Lipschitz balls. Dropping the usual assumption that the density is bounded away from zero, we obtain the minimax rates (nln n)-s/(s+d) + n-1/2 for 0<s≤ 2 for densities supported on [0,1]d, where s is the smoothness parameter and n is the number of independent samples. We generalize the results to densities with unbounded support: given an Orlicz functions Psi of rapid growth (such as the sub-exponential and sub-Gaussian classes), the minimax rates for densities with bounded Psi-Orlicz norm increase to (nln n)-s/(s+d) (Psi-1(n))d(1-d/p(s+d)) + n-1/2, where p is the norm parameter in the Lipschitz ball. We also show that the integral-form plug-in estimators with kernel density estimates fail to achieve the minimax rates, and characterize their worst case performances over the Lipschitz ball. One of the key steps in analyzing the bias relies on a novel application of the Hardy-Littlewood maximal inequality, which also leads to a new inequality on the Fisher information that may be of independent interest.