# Information Theory Measures via Multidimensional Gaussianization

arXiv: https://arxiv.org/abs/2010.03807 · HTML: https://mlsanity.com/paper/2010.03807/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2020.

## Built on by (most cited first, 5 of 5)

- [Looks Too Good To Be True: An Information-Theoretic Analysis of Hallucinations in Generative Restoration Models](https://mlsanity.com/paper/2405.16475/) (2024)
- [OceanBench: The Sea Surface Height Edition](https://mlsanity.com/paper/2309.15599/) (2023)
- [Unsupervised Anomaly and Change Detection with Multivariate Gaussianization](https://mlsanity.com/paper/2204.05699/) (2022)
- [Orthonormal Convolutions for the Rotation Based Iterative Gaussianization](https://mlsanity.com/paper/2206.03860/) (2022)
- [Image Statistics Predict the Sensitivity of Perceptual Quality Metrics](https://mlsanity.com/paper/2303.09874/) (2023)

## Abstract (arXiv)

Information theory is an outstanding framework to measure uncertainty, dependence and relevance in data and systems. It has several desirable properties for real world applications: it naturally deals with multivariate data, it can handle heterogeneous data types, and the measures can be interpreted in physical units. However, it has not been adopted by a wider audience because obtaining information from multidimensional data is a challenging problem due to the curse of dimensionality. Here we propose an indirect way of computing information based on a multivariate Gaussianization transform. Our proposal mitigates the difficulty of multivariate density estimation by reducing it to a composition of tractable (marginal) operations and simple linear transformations, which can be interpreted as a particular deep neural network. We introduce specific Gaussianization-based methodologies to estimate total correlation, entropy, mutual information and Kullback-Leibler divergence. We compare them to recent estimators showing the accuracy on synthetic data generated from different multivariate distributions. We made the tools and datasets publicly available to provide a test-bed to analyze future methodologies. Results show that our proposal is superior to previous estimators particularly in high-dimensional scenarios; and that it leads to interesting insights in neuroscience, geoscience, computer vision, and machine learning.