# A Deep Bayesian Nonparametric Framework for Robust Mutual Information Estimation

arXiv: https://arxiv.org/abs/2503.08902 · HTML: https://mlsanity.com/paper/2503.08902/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 30 of 30)

- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Adversarial Feature Learning](https://mlsanity.com/paper/1605.09782/) (2016)
- [Generative Moment Matching Networks](https://mlsanity.com/paper/1502.02761/) (2015)
- [Adversarially Learned Inference](https://mlsanity.com/paper/1606.00704/) (2016)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Generation of 3D Brain MRI Using Auto-Encoding Generative Adversarial Networks](https://mlsanity.com/paper/1908.02498/) (2019)
- [Variational Approaches for Auto-Encoding Generative Adversarial Networks](https://mlsanity.com/paper/1706.04987/) (2017)
- [Robust Bayesian Inference for Simulator-based Models via the MMD Posterior Bootstrap](https://mlsanity.com/paper/2202.04744/) (2022)
- [General Bayesian Updating and the Loss-Likelihood Bootstrap](https://mlsanity.com/paper/1709.07616/) (2017)
- [Self-supervised Representation Learning with Relative Predictive Coding](https://mlsanity.com/paper/2103.11275/) (2021)
- [Nonparametric independence testing via mutual information](https://mlsanity.com/paper/1711.06642/) (2017)
- [Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization](https://mlsanity.com/paper/2107.01131/) (2021)
- [Nonparametric learning from Bayesian models with randomized objective functions](https://mlsanity.com/paper/1806.11544/) (2018)
- [Scalable Nonparametric Sampling from Multimodal Posteriors with the Posterior Bootstrap](https://mlsanity.com/paper/1902.03175/) (2019)
- [Mutual Information Gradient Estimation for Representation Learning](https://mlsanity.com/paper/2005.01123/) (2020)
- [Enhancing Transfer Learning with Flexible Nonparametric Posterior Sampling](https://mlsanity.com/paper/2403.07282/) (2024)
- [A Semi-Bayesian Nonparametric Estimator of the Maximum Mean Discrepancy Measure: Applications in Goodness-of-Fit Testing and Generative Adversarial Networks](https://mlsanity.com/paper/2303.02637/) (2023)
- [A Test for Independence Via Bayesian Nonparametric Estimation of Mutual Information](https://mlsanity.com/paper/2002.03490/) (2020)
- [Improved α-GAN architecture for generating 3D connected volumes with an application to radiosurgery treatment planning](https://mlsanity.com/paper/2207.11223/) (2022)
- [A Bayesian Non-parametric Approach to Generative Models: Integrating Variational Autoencoder and Generative Adversarial Networks using Wasserstein and Maximum Mean Discrepancy](https://mlsanity.com/paper/2308.14048/) (2023)
- [Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimization](https://mlsanity.com/paper/2401.15771/) (2024)

## Abstract (arXiv)

Mutual Information (MI) is a crucial measure for capturing dependencies between variables, but exact computation is challenging in high dimensions with intractable likelihoods, impacting accuracy and robustness. One idea is to use an auxiliary neural network to train an MI estimator; however, methods based on the empirical distribution function (EDF) can introduce sharp fluctuations in the MI loss due to poor out-of-sample performance, destabilizing convergence. We present a Bayesian nonparametric (BNP) solution for training an MI estimator by constructing the MI loss with a finite representation of the Dirichlet process posterior to incorporate regularization in the training process. With this regularization, the MI loss integrates both prior knowledge and empirical data to reduce the loss sensitivity to fluctuations and outliers in the sample data, especially in small sample settings like mini-batches. This approach addresses the challenge of balancing accuracy and low variance by effectively reducing variance, leading to stabilized and robust MI loss gradients during training and enhancing the convergence of the MI approximation while offering stronger theoretical guarantees for convergence. We explore the application of our estimator in maximizing MI between the data space and the latent space of a variational autoencoder. Experimental results demonstrate significant improvements in convergence over EDF-based methods, with applications across synthetic and real datasets, notably in 3D CT image generation, yielding enhanced structure discovery and reduced overfitting in data synthesis. While this paper focuses on generative models in application, the proposed estimator is not restricted to this setting and can be applied more broadly in various BNP learning procedures.