# CCMI : Classifier based Conditional Mutual Information Estimation

arXiv: https://arxiv.org/abs/1906.01824 · HTML: https://mlsanity.com/paper/1906.01824/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2019.

## Built on by (most cited first, 30 of 37)

- [HDMI: High-order Deep Multiplex Infomax](https://mlsanity.com/paper/2102.07810/) (2021)
- [Factorized Contrastive Learning: Going Beyond Multi-view Redundancy](https://mlsanity.com/paper/2306.05268/) (2023)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration](https://mlsanity.com/paper/2109.14591/) (2021)
- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [Conditional Mutual Information for Disentangled Representations in Reinforcement Learning](https://mlsanity.com/paper/2305.14133/) (2023)
- [Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities](https://mlsanity.com/paper/2602.05073/) (2026)
- [InfoFair: Information-Theoretic Intersectional Fairness](https://mlsanity.com/paper/2105.11069/) (2021)
- [Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularization](https://mlsanity.com/paper/2103.04813/) (2021)
- [A Causal Adjustment Module for Debiasing Scene Graph Generation](https://mlsanity.com/paper/2503.17862/) (2025)
- [Out-of-Distribution Generalization Analysis via Influence Function](https://mlsanity.com/paper/2101.08521/) (2021)
- [Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learning](https://mlsanity.com/paper/2106.00252/) (2021)
- [Diffeomorphic Information Neural Estimation](https://mlsanity.com/paper/2211.10856/) (2022)
- [Max-Sliced Mutual Information](https://mlsanity.com/paper/2309.16200/) (2023)
- [Conditional Independence Testing via Latent Representation Learning](https://mlsanity.com/paper/2209.01547/) (2022)
- [Disentanglement and Generalization Under Correlation Shifts](https://mlsanity.com/paper/2112.14754/) (2021)
- [SemCal: Semantic LiDAR-Camera Calibration using Neural MutualInformation Estimator](https://mlsanity.com/paper/2109.10270/) (2021)
- [CDANs: Temporal Causal Discovery from Autocorrelated and Non-Stationary Time Series Data](https://mlsanity.com/paper/2302.03246/) (2023)
- [Can Information Flows Suggest Targets for Interventions in Neural Circuits?](https://mlsanity.com/paper/2111.05299/) (2021)
- [Nearest-Neighbor Sampling Based Conditional Independence Testing](https://mlsanity.com/paper/2304.04183/) (2023)
- [CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systems](https://mlsanity.com/paper/2406.17300/) (2024)
- [Efficient Ensemble Conditional Independence Test Framework for Causal Discovery](https://mlsanity.com/paper/2509.21021/) (2025)
- [Expression Syntax Information Bottleneck for Math Word Problems](https://mlsanity.com/paper/2310.15664/) (2023)
- [Neural Mutual Information Estimation with Vector Copulas](https://mlsanity.com/paper/2510.20968/) (2025)
- [Causal Discovery via Conditional Independence Testing with Proxy Variables](https://mlsanity.com/paper/2305.05281/) (2023)
- [Quantifying Feature Contributions to Overall Disparity Using Information Theory](https://mlsanity.com/paper/2206.08454/) (2022)
- [TREET: TRansfer Entropy Estimation via Transformers](https://mlsanity.com/paper/2402.06919/) (2024)
- [Which bits went where? Past and future transfer entropy decomposition with the information bottleneck](https://mlsanity.com/paper/2411.04992/) (2024)
- [Calibrating LiDAR and Camera using Semantic Mutual information](https://mlsanity.com/paper/2104.12023/) (2021)
- [MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories](https://mlsanity.com/paper/2106.01808/) (2021)

## Abstract (arXiv)

Conditional Mutual Information (CMI) is a measure of conditional dependence between random variables X and Y, given another random variable Z. It can be used to quantify conditional dependence among variables in many data-driven inference problems such as graphical models, causal learning, feature selection and time-series analysis. While k-nearest neighbor (kNN) based estimators as well as kernel-based methods have been widely used for CMI estimation, they suffer severely from the curse of dimensionality. In this paper, we leverage advances in classifiers and generative models to design methods for CMI estimation. Specifically, we introduce an estimator for KL-Divergence based on the likelihood ratio by training a classifier to distinguish the observed joint distribution from the product distribution. We then show how to construct several CMI estimators using this basic divergence estimator by drawing ideas from conditional generative models. We demonstrate that the estimates from our proposed approaches do not degrade in performance with increasing dimension and obtain significant improvement over the widely used KSG estimator. Finally, as an application of accurate CMI estimation, we use our best estimator for conditional independence testing and achieve superior performance than the state-of-the-art tester on both simulated and real data-sets.