# Improving Numerical Stability of Normalized Mutual Information Estimator on High Dimensions

arXiv: https://arxiv.org/abs/2410.07642 · HTML: https://mlsanity.com/paper/2410.07642/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 48 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Builds on (most cited first, 7 of 7)

- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Beyond Normal: On the Evaluation of Mutual Information Estimators](https://mlsanity.com/paper/2306.11078/) (2023)
- [Predictive information in a sensory population](https://mlsanity.com/paper/1307.0225/) (2013)
- [On the Properties and Estimation of Pointwise Mutual Information Profiles](https://mlsanity.com/paper/2310.10240/) (2023)
- [Data-Efficient Mutual Information Neural Estimator](https://mlsanity.com/paper/1905.03319/) (2019)

## Built on by (most cited first, 1 of 1)

- [NMINE: Normalized Mutual Information Neural Estimation](https://mlsanity.com/paper/2607.27710/) (2026)

## Abstract (arXiv)

Mutual information provides a powerful, general-purpose metric for quantifying the amount of shared information between variables. Estimating normalized mutual information using a k-Nearest Neighbor (k-NN) based approach involves the calculation of the scaling-invariant k-NN radius. Calculation of the radius suffers from numerical overflow when the joint dimensionality of the data becomes high, typically in the range of several hundred dimensions. To address this issue, we propose a logarithmic transformation technique that improves the numerical stability of the radius calculation in high-dimensional spaces. By applying the proposed transformation during the calculation of the radius, numerical overflow is avoided, and precision is maintained. Proposed transformation is validated through both theoretical analysis and empirical evaluation, demonstrating its ability to stabilize the calculation without compromising precision, increasing bias, or adding significant computational overhead, while also helping to maintain estimator variance.