# Estimating Mutual Information for Discrete-Continuous Mixtures

arXiv: https://arxiv.org/abs/1709.06212 · HTML: https://mlsanity.com/paper/1709.06212/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2017.

## Built on by (most cited first, 30 of 34)

- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search](https://mlsanity.com/paper/2109.05433/) (2021)
- [Beyond Normal: On the Evaluation of Mutual Information Estimators](https://mlsanity.com/paper/2306.11078/) (2023)
- [A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition](https://mlsanity.com/paper/2105.04187/) (2021)
- [Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group](https://mlsanity.com/paper/2107.06898/) (2021)
- [Quantifying the redundancy between prosody and text](https://mlsanity.com/paper/2311.17233/) (2023)
- [On the Effectiveness of Hybrid Mutual Information Estimation](https://mlsanity.com/paper/2306.00608/) (2023)
- [On the Properties and Estimation of Pointwise Mutual Information Profiles](https://mlsanity.com/paper/2310.10240/) (2023)
- [Can Information Flows Suggest Targets for Interventions in Neural Circuits?](https://mlsanity.com/paper/2111.05299/) (2021)
- [Fast Rate Generalization Error Bounds: Variations on a Theme](https://mlsanity.com/paper/2205.03131/) (2022)
- [Fast Rate Information-theoretic Bounds on Generalization Errors](https://mlsanity.com/paper/2303.14658/) (2023)
- [MITFAS: Mutual Information based Temporal Feature Alignment and Sampling for Aerial Video Action Recognition](https://mlsanity.com/paper/2303.02575/) (2023)
- [Expected information gain estimation via density approximations: Sample allocation and dimension reduction](https://mlsanity.com/paper/2411.08390/) (2024)
- [ARLO: A Framework for Automated Reinforcement Learning](https://mlsanity.com/paper/2205.10416/) (2022)
- [Greedy Modality Selection via Approximate Submodular Maximization](https://mlsanity.com/paper/2210.12562/) (2022)
- [Causal Feature Selection via Transfer Entropy](https://mlsanity.com/paper/2310.11059/) (2023)
- [State and Action Factorization in Power Grids](https://mlsanity.com/paper/2409.04467/) (2024)
- [Fairness Overfitting in Machine Learning: An Information-Theoretic Perspective](https://mlsanity.com/paper/2506.07861/) (2025)
- [MMG: Mutual Information Estimation via the MMSE Gap in Diffusion](https://mlsanity.com/paper/2509.20609/) (2025)
- [Stochastic Mutual Information Gradient Estimation for Dimensionality Reduction Networks](https://mlsanity.com/paper/2105.00191/) (2021)
- [Trying to Outrun Causality with Machine Learning: Limitations of Model Explainability Techniques for Identifying Predictive Variables](https://mlsanity.com/paper/2202.09875/) (2022)
- [Improving Mutual Information based Feature Selection by Boosting Unique Relevance](https://mlsanity.com/paper/2212.06143/) (2022)
- [Multi-Group Proportional Representation in Retrieval](https://mlsanity.com/paper/2407.08571/) (2024)
- [Probabilistic Variational Contrastive Learning](https://mlsanity.com/paper/2506.10159/) (2025)
- [Non-parametric Conditional Independence Testing for Mixed Continuous-Categorical Variables: A Novel Method and Numerical Evaluation](https://mlsanity.com/paper/2310.11132/) (2023)
- [Generalizing Supervised Contrastive learning: A Projection Perspective](https://mlsanity.com/paper/2506.09810/) (2025)
- [Multimodal Datasets with Controllable Mutual Information](https://mlsanity.com/paper/2510.21686/) (2025)
- [Beyond Voice Identity Conversion: Manipulating Voice Attributes by Adversarial Learning of Structured Disentangled Representations](https://mlsanity.com/paper/2107.12346/) (2021)
- [Implicit Feature Decoupling with Depthwise Quantization](https://mlsanity.com/paper/2203.08080/) (2022)
- [Pitfalls of Climate Network Construction: A Statistical Perspective](https://mlsanity.com/paper/2211.02888/) (2022)

## Abstract (arXiv)

Estimating mutual information from observed samples is a basic primitive, useful in several machine learning tasks including correlation mining, information bottleneck clustering, learning a Chow-Liu tree, and conditional independence testing in (causal) graphical models. While mutual information is a well-defined quantity in general probability spaces, existing estimators can only handle two special cases of purely discrete or purely continuous pairs of random variables. The main challenge is that these methods first estimate the (differential) entropies of X, Y and the pair (X;Y) and add them up with appropriate signs to get an estimate of the mutual information. These 3H-estimators cannot be applied in general mixture spaces, where entropy is not well-defined. In this paper, we design a novel estimator for mutual information of discrete-continuous mixtures. We prove that the proposed estimator is consistent. We provide numerical experiments suggesting superiority of the proposed estimator compared to other heuristics of adding small continuous noise to all the samples and applying standard estimators tailored for purely continuous variables, and quantizing the samples and applying standard estimators tailored for purely discrete variables. This significantly widens the applicability of mutual information estimation in real-world applications, where some variables are discrete, some continuous, and others are a mixture between continuous and discrete components.