mlsanityData to 7 Oct 2026

On the Properties and Estimation of Pointwise Mutual Information Profiles

arXiv 2310.10240 ↗First posted 16 Oct 2023stat.ML cs.IT cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

The pointwise mutual information profile, or simply profile, is the distribution of pointwise mutual information for a given pair of random variables. One of its important properties is that its expected value is precisely the mutual information between these random variables. In this paper, we analytically describe the profiles of multivariate normal distributions and introduce a novel family of distributions, Bend and Mix Models, for which the profile can be accurately estimated using Monte Carlo methods. We then show how Bend and Mix Models can be used to study the limitations of existing mutual information estimators, investigate the behavior of neural critics used in variational estimators, and understand the effect of experimental outliers on mutual information estimation. Finally, we show how Bend and Mix Models can be used to obtain model-based Bayesian estimates of mutual information, suitable for problems with available domain expertise in which uncertainty quantification is necessary.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 24 of its 63 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

24 of 63 references.

References · 24

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  3. 2018MINE: Mutual Information Neural Estimationcited by 680
  4. 2019On Variational Bounds of Mutual Informationcited by 550
  5. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  6. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  7. 2019Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyrocited by 187
  8. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  9. 2017TensorFlow Distributionscited by 105
  10. 2014Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing Itcited by 92
  11. 2019Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMCcited by 90
  12. 2017On Nesting Monte Carlo Estimatorscited by 82
12 more
  1. 2019Analogies Explained: Towards Understanding Word Embeddingscited by 63
  2. 2023Beyond Normal: On the Evaluation of Mutual Information Estimatorscited by 37
  3. 2017Estimating Mixture Entropy with Pairwise Distancescited by 34
  4. 2017Estimating Mutual Information for Discrete-Continuous Mixturescited by 34
  5. 2018Revisiting the Gelman-Rubin Diagnosticcited by 18
  6. 2018Nonparametric learning from Bayesian models with randomized objective functionscited by 17
  7. 2020SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Modelscited by 14
  8. 2014Approximate Models and Robust Decisionscited by 10
  9. 2015Comparison of Bayesian predictive methods for model selectioncited by 10
  10. 2019Testing Conditional Independence on Discrete Data using Stochastic Complexitycited by 10
  11. 2018A whitening approach to probabilistic canonical correlation analysis for omics data integrationcited by 6
  12. 2022Generative Models: An Interdisciplinary Perspectivecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2025InfoBridge: Mutual Information estimation via Bridge Matchingcited by 7
  2. 2024Improving Numerical Stability of Normalized Mutual Information Estimator on High Dimensionscited by 1
  3. 2025Multimodal Datasets with Controllable Mutual Informationcited by 1
  4. 2026InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimatecited by 1
  5. 2024Mutual Information Multinomial Estimationcited by 0
  6. 2025Towards a pretrained deep learning estimator of the Linfoot informational correlationcited by 0
  7. 2026Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimationcited by 0