# On the Properties and Estimation of Pointwise Mutual Information Profiles

arXiv: https://arxiv.org/abs/2310.10240 · HTML: https://mlsanity.com/paper/2310.10240/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 63 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro](https://mlsanity.com/paper/1912.11554/) (2019)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [TensorFlow Distributions](https://mlsanity.com/paper/1711.10604/) (2017)
- [Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It](https://mlsanity.com/paper/1412.3730/) (2014)
- [Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMC](https://mlsanity.com/paper/1903.08008/) (2019)
- [On Nesting Monte Carlo Estimators](https://mlsanity.com/paper/1709.06181/) (2017)
- [Analogies Explained: Towards Understanding Word Embeddings](https://mlsanity.com/paper/1901.09813/) (2019)
- [Beyond Normal: On the Evaluation of Mutual Information Estimators](https://mlsanity.com/paper/2306.11078/) (2023)
- [Estimating Mixture Entropy with Pairwise Distances](https://mlsanity.com/paper/1706.02419/) (2017)
- [Estimating Mutual Information for Discrete-Continuous Mixtures](https://mlsanity.com/paper/1709.06212/) (2017)
- [Revisiting the Gelman-Rubin Diagnostic](https://mlsanity.com/paper/1812.09384/) (2018)
- [Nonparametric learning from Bayesian models with randomized objective functions](https://mlsanity.com/paper/1806.11544/) (2018)
- [SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models](https://mlsanity.com/paper/2004.00353/) (2020)
- [Approximate Models and Robust Decisions](https://mlsanity.com/paper/1402.6118/) (2014)
- [Comparison of Bayesian predictive methods for model selection](https://mlsanity.com/paper/1503.08650/) (2015)
- [Testing Conditional Independence on Discrete Data using Stochastic Complexity](https://mlsanity.com/paper/1903.04829/) (2019)
- [A whitening approach to probabilistic canonical correlation analysis for omics data integration](https://mlsanity.com/paper/1802.03490/) (2018)
- [Generative Models: An Interdisciplinary Perspective](https://mlsanity.com/paper/2208.06011/) (2022)

## Built on by (most cited first, 7 of 7)

- [InfoBridge: Mutual Information estimation via Bridge Matching](https://mlsanity.com/paper/2502.01383/) (2025)
- [Improving Numerical Stability of Normalized Mutual Information Estimator on High Dimensions](https://mlsanity.com/paper/2410.07642/) (2024)
- [Multimodal Datasets with Controllable Mutual Information](https://mlsanity.com/paper/2510.21686/) (2025)
- [InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate](https://mlsanity.com/paper/2606.00241/) (2026)
- [Mutual Information Multinomial Estimation](https://mlsanity.com/paper/2408.09377/) (2024)
- [Towards a pretrained deep learning estimator of the Linfoot informational correlation](https://mlsanity.com/paper/2512.12358/) (2025)
- [Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation](https://mlsanity.com/paper/2607.03487/) (2026)

## Abstract (arXiv)

The pointwise mutual information profile, or simply profile, is the distribution of pointwise mutual information for a given pair of random variables. One of its important properties is that its expected value is precisely the mutual information between these random variables. In this paper, we analytically describe the profiles of multivariate normal distributions and introduce a novel family of distributions, Bend and Mix Models, for which the profile can be accurately estimated using Monte Carlo methods. We then show how Bend and Mix Models can be used to study the limitations of existing mutual information estimators, investigate the behavior of neural critics used in variational estimators, and understand the effect of experimental outliers on mutual information estimation. Finally, we show how Bend and Mix Models can be used to obtain model-based Bayesian estimates of mutual information, suitable for problems with available domain expertise in which uncertainty quantification is necessary.