# Mutual Information Gradient Estimation for Representation Learning

arXiv: https://arxiv.org/abs/2005.01123 · HTML: https://mlsanity.com/paper/2005.01123/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2020.

## Built on by (most cited first, 14 of 14)

- [Provable Dynamic Fusion for Low-Quality Multimodal Data](https://mlsanity.com/paper/2306.02050/) (2023)
- [Self-Supervision Can Be a Good Few-Shot Learner](https://mlsanity.com/paper/2207.09176/) (2022)
- [Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization](https://mlsanity.com/paper/2107.01131/) (2021)
- [A Quantitative Metric for Privacy Leakage in Federated Learning](https://mlsanity.com/paper/2102.13472/) (2021)
- [Beyond Redundancy: Information-aware Unsupervised Multiplex Graph Structure Learning](https://mlsanity.com/paper/2409.17386/) (2024)
- [InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactions](https://mlsanity.com/paper/2509.25270/) (2025)
- [The Hidden Link Between RLHF and Contrastive Learning](https://mlsanity.com/paper/2506.22578/) (2025)
- [MVEB: Self-Supervised Learning with Multi-View Entropy Bottleneck](https://mlsanity.com/paper/2403.19078/) (2024)
- [AutoTransfer: Subject Transfer Learning with Censored Representations on Biosignals Data](https://mlsanity.com/paper/2112.09796/) (2021)
- [Cross-modal Image Retrieval with Deep Mutual Information Maximization](https://mlsanity.com/paper/2103.06032/) (2021)
- [Stochastic Mutual Information Gradient Estimation for Dimensionality Reduction Networks](https://mlsanity.com/paper/2105.00191/) (2021)
- [AI-based association analysis for medical imaging using latent-space geometric confounder correction](https://mlsanity.com/paper/2311.12836/) (2023)
- [Privacy for Fairness: Information Obfuscation for Fair Representation Learning with Local Differential Privacy](https://mlsanity.com/paper/2402.10473/) (2024)
- [A Deep Bayesian Nonparametric Framework for Robust Mutual Information Estimation](https://mlsanity.com/paper/2503.08902/) (2025)

## Abstract (arXiv)

Mutual Information (MI) plays an important role in representation learning. However, MI is unfortunately intractable in continuous and high-dimensional settings. Recent advances establish tractable and scalable MI estimators to discover useful representation. However, most of the existing methods are not capable of providing an accurate estimation of MI with low-variance when the MI is large. We argue that directly estimating the gradients of MI is more appealing for representation learning than estimating MI in itself. To this end, we propose the Mutual Information Gradient Estimator (MIGE) for representation learning based on the score estimation of implicit distributions. MIGE exhibits a tight and smooth gradient estimation of MI in the high-dimensional and large-MI settings. We expand the applications of MIGE in both unsupervised learning of deep representations based on InfoMax and the Information Bottleneck method. Experimental results have indicated significant performance improvement in learning useful representation.