# Efficient multivariate entropy estimation via k-nearest neighbour distances

arXiv: https://arxiv.org/abs/1606.00304 · HTML: https://mlsanity.com/paper/1606.00304/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2016.

## Built on by (most cited first, 28 of 28)

- [Mutual Information Estimation via Normalizing Flows](https://mlsanity.com/paper/2403.02187/) (2024)
- [A Differential Entropy Estimator for Training Neural Networks](https://mlsanity.com/paper/2202.06618/) (2022)
- [Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression](https://mlsanity.com/paper/2305.08013/) (2023)
- [Non-Asymptotic Performance Guarantees for Neural Estimation of mathsff-Divergences](https://mlsanity.com/paper/2103.06923/) (2021)
- [The Fundamental Limits of Structure-Agnostic Functional Estimation](https://mlsanity.com/paper/2305.04116/) (2023)
- [Structure Learning for Directed Trees](https://mlsanity.com/paper/2108.08871/) (2021)
- [Sourcerer: Sample-based Maximum Entropy Source Distribution Estimation](https://mlsanity.com/paper/2402.07808/) (2024)
- [Exploring Exploration in Bayesian Optimization](https://mlsanity.com/paper/2502.08208/) (2025)
- [Learning of Population Dynamics: Inverse Optimization Meets JKO Scheme](https://mlsanity.com/paper/2506.01502/) (2025)
- [InfoBridge: Mutual Information estimation via Bridge Matching](https://mlsanity.com/paper/2502.01383/) (2025)
- [Nearest Neighbor Sampling for Covariate Shift Adaptation](https://mlsanity.com/paper/2312.09969/) (2023)
- [Explaining Grokking and Information Bottleneck through Neural Collapse Emergence](https://mlsanity.com/paper/2509.20829/) (2025)
- [Limit theorems of Chatterjee's rank correlation](https://mlsanity.com/paper/2204.08031/) (2022)
- [Conditional Mean and Variance Estimation via textitk-NN Algorithm with Automated Variance Selection](https://mlsanity.com/paper/2402.01635/) (2024)
- [A Flexible Approach for Normal Approximation of Geometric and Topological Statistics](https://mlsanity.com/paper/2210.10744/) (2022)
- [MIST: Mutual Information Estimation Via Supervised Training](https://mlsanity.com/paper/2511.18945/) (2025)
- [Nearest-Neighbor Density Estimation for Dependency Suppression](https://mlsanity.com/paper/2603.04224/) (2026)
- [Order-based Rehearsal Learning](https://mlsanity.com/paper/2605.04955/) (2026)
- [Manifold Sampling via Entropy Maximization](https://mlsanity.com/paper/2605.12338/) (2026)
- [Boosting Reservoir Computing with Brain-inspired Adaptive Dynamics](https://mlsanity.com/paper/2504.12480/) (2025)
- [Nonparametric Estimation of Joint Entropy via Partitioned Sample-Spacing](https://mlsanity.com/paper/2511.13602/) (2025)
- [Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control](https://mlsanity.com/paper/2512.24826/) (2025)
- [Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation](https://mlsanity.com/paper/2607.03487/) (2026)
- [Graph Memory: Spectral Associative Memory via Dirichlet Energy](https://mlsanity.com/paper/2609.32365/) (2026)
- [A Hierarchy of Entropy-Shapley Games for Multivariate Predictive Uncertainty](https://mlsanity.com/paper/2609.35217/) (2026)
- [Local polynomial density ratio estimation](https://mlsanity.com/paper/2609.38412/) (2026)
- [From Pixels to Privacy-Sensitive Features: An Information-Theoretic Framework for Reconstruction Bounds](https://mlsanity.com/paper/2610.13768/) (2026)
- [Copula Active Subspaces II: Error Decomposition, A Posteriori Estimation, and Sharpness of the Bounds](https://mlsanity.com/paper/2610.14498/) (2026)

## Abstract (arXiv)

Many statistical procedures, including goodness-of-fit tests and methods for independent component analysis, rely critically on the estimation of the entropy of a distribution. In this paper, we seek entropy estimators that are efficient and achieve the local asymptotic minimax lower bound with respect to squared error loss. To this end, we study weighted averages of the estimators originally proposed by Kozachenko and Leonenko (1987), based on the k-nearest neighbour distances of a sample of n independent and identically distributed random vectors in ℝd. A careful choice of weights enables us to obtain an efficient estimator in arbitrary dimensions, given sufficient smoothness, while the original unweighted estimator is typically only efficient when d ≤ 3. In addition to the new estimator proposed and theoretical understanding provided, our results facilitate the construction of asymptotically valid confidence intervals for the entropy of asymptotically minimal width.