# MIND: Inductive Mutual Information Estimation, A Convex Maximum-Entropy Copula Approach

arXiv: https://arxiv.org/abs/2102.13182 · HTML: https://mlsanity.com/paper/2102.13182/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 25 Feb 2021.

## Builds on (most cited first, 18 of 18)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Adversarially Learned Inference](https://mlsanity.com/paper/1606.00704/) (2016)
- [Unrolled Generative Adversarial Networks](https://mlsanity.com/paper/1611.02163/) (2016)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Mode Regularized Generative Adversarial Networks](https://mlsanity.com/paper/1612.02136/) (2016)
- [PacGAN: The power of two samples in generative adversarial networks](https://mlsanity.com/paper/1712.04086/) (2017)
- [Dual Discriminator Generative Adversarial Nets](https://mlsanity.com/paper/1709.03831/) (2017)
- [Efficient Estimation of Mutual Information for Strongly Dependent Variables](https://mlsanity.com/paper/1411.2003/) (2014)
- [Bayesian GAN](https://mlsanity.com/paper/1705.09558/) (2017)
- [Scalable Mutual Information Estimation using Dependence Graphs](https://mlsanity.com/paper/1801.09125/) (2018)

## Built on by (most cited first, 6 of 6)

- [Diffeomorphic Information Neural Estimation](https://mlsanity.com/paper/2211.10856/) (2022)
- [Your copula is a classifier in disguise: classification-based copula density estimation](https://mlsanity.com/paper/2411.03014/) (2024)
- [Inference and Sampling for Archimax Copulas](https://mlsanity.com/paper/2205.14025/) (2022)
- [Neural Mutual Information Estimation with Vector Copulas](https://mlsanity.com/paper/2510.20968/) (2025)
- [LeanML: A Design Pattern To Slash Avoidable Wastes in Machine Learning Projects](https://mlsanity.com/paper/2107.08066/) (2021)
- [Amortized Conditional Independence Testing](https://mlsanity.com/paper/2502.20925/) (2025)

## Abstract (arXiv)

We propose a novel estimator of the mutual information between two ordinal vectors x and y. Our approach is inductive (as opposed to deductive) in that it depends on the data generating distribution solely through some nonparametric properties revealing associations in the data, and does not require having enough data to fully characterize the true joint distributions Px, y. Specifically, our approach consists of (i) noting that Ileft(y; xright) = Ileft(uy; uxright) where uy and ux are the copula-uniform dual representations of y and x (i.e. their images under the probability integral transform), and (ii) estimating the copula entropies hleft(uyright), hleft(uxright) and hleft(uy, uxright) by solving a maximum-entropy problem over the space of copula densities under a constraint of the type αm = Eleft[φm(uy, ux)right]. We prove that, so long as the constraint is feasible, this problem admits a unique solution, it is in the exponential family, and it can be learned by solving a convex optimization problem. The resulting estimator, which we denote MIND, is marginal-invariant, always non-negative, unbounded for any sample size n, consistent, has MSE rate O(1/n), and is more data-efficient than competing approaches. Beyond mutual information estimation, we illustrate that our approach may be used to mitigate mode collapse in GANs by maximizing the entropy of the copula of fake samples, a model we refer to as Copula Entropy Regularized GAN (CER-GAN).