mlsanityData to 7 Oct 2026

Mutual Information Multinomial Estimation

arXiv 2408.09377 ↗First posted 18 Aug 2024cs.LG cs.IT math.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

Estimating mutual information (MI) is a fundamental yet challenging task in data science and machine learning. This work proposes a new estimator for mutual information. Our main discovery is that a preliminary estimate of the data distribution can dramatically help estimate. This preliminary estimate serves as a bridge between the joint and the marginal distribution, and by comparing with this bridge distribution we can easily obtain the true difference between the joint distributions and the marginal distributions. Experiments on diverse tasks including non-Gaussian synthetic problems with known ground-truth and real-world applications demonstrate the advantages of our method.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 66 references that are in the map. First posted on arXiv 18 Aug 2024.

Builds on

31 of 66 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2022Flow Matching for Generative Modelingcited by 5,426
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2016Deep Variational Information Bottleneckcited by 915
  7. 2019On Variational Bounds of Mutual Informationcited by 550
  8. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  9. 2019On Mutual Information Maximization for Representation Learningcited by 315
  10. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 233
  11. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  12. 2019Variational Bayesian Optimal Experimental Designcited by 116
19 more
  1. 2021Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Designcited by 103
  2. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  3. 2020Neural Approximate Sufficient Statistics for Implicit Modelscited by 72
  4. 2021Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoodscited by 61
  5. 2021Density Ratio Estimation via Infinitesimal Classificationcited by 59
  6. 2020On Mutual Information in Contrastive Learning for Visual Representationscited by 54
  7. 2014Efficient Estimation of Mutual Information for Strongly Dependent Variablescited by 46
  8. 2023Beyond Normal: On the Evaluation of Mutual Information Estimatorscited by 37
  9. 2021Featurized Density Ratio Estimationcited by 33
  10. 2023MINDE: Mutual Information Neural Diffusion Estimationcited by 31
  11. 2024Mutual Information Estimation via Normalizing Flowscited by 22
  12. 2013High-Dimensional Probability Estimation with Deep Density Modelscited by 17
  13. 2023Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regressioncited by 17
  14. 2023Max-Sliced Mutual Informationcited by 10
  15. 2024Poly-View Contrastive Learningcited by 9
  16. 2023Scalable Infomin Learningcited by 8
  17. 2023On the Properties and Estimation of Pointwise Mutual Information Profilescited by 7
  18. 2021A Unified Framework for Multi-distribution Density Ratio Estimationcited by 5
  19. 2024Overcoming Saturation in Density Ratio Estimation by Iterated Regularizationcited by 5

Built on by

Not cited yet.