# Mutual Information Estimation via f-Divergence and Data Derangements

arXiv: https://arxiv.org/abs/2305.20025 · HTML: https://mlsanity.com/paper/2305.20025/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 33 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 13 of 13)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Copula Density Neural Estimation](https://mlsanity.com/paper/2211.15353/) (2022)
- [Neural Joint Entropy Estimation](https://mlsanity.com/paper/2012.11197/) (2020)
- [MIND: Maximum Mutual Information Based Neural Decoder](https://mlsanity.com/paper/2205.07061/) (2022)
- [Cooperative Channel Capacity Learning](https://mlsanity.com/paper/2305.13493/) (2023)

## Built on by (most cited first, 17 of 17)

- [MINDE: Mutual Information Neural Diffusion Estimation](https://mlsanity.com/paper/2310.09031/) (2023)
- [f-Divergence Based Classification: Beyond the Use of Cross-Entropy](https://mlsanity.com/paper/2401.01268/) (2024)
- [InfoBridge: Mutual Information estimation via Bridge Matching](https://mlsanity.com/paper/2502.01383/) (2025)
- [Generative AI collective behavior needs an interactionist paradigm](https://mlsanity.com/paper/2601.10567/) (2026)
- [Connecting Jensen-Shannon and Kullback-Leibler Divergences: A New Bound for Representation Learning](https://mlsanity.com/paper/2510.20644/) (2025)
- [Neural Mutual Information Estimation with Vector Copulas](https://mlsanity.com/paper/2510.20968/) (2025)
- [Robust Classification with Noisy Labels Based on Posterior Maximization](https://mlsanity.com/paper/2504.06805/) (2025)
- [Accurate Estimation of Mutual Information in High Dimensional Data](https://mlsanity.com/paper/2506.00330/) (2025)
- [A Unified Framework for Diffusion Model Unlearning with f-Divergence](https://mlsanity.com/paper/2509.21167/) (2025)
- [Contrastive Predictive Coding Done Right for Mutual Information Estimation](https://mlsanity.com/paper/2510.25983/) (2025)
- [Feedback Lunch: Learned Feedback Codes for Secure Communications](https://mlsanity.com/paper/2510.16620/) (2025)
- [Discrete Bridges for Mutual Information Estimation](https://mlsanity.com/paper/2602.08894/) (2026)
- [One-Step Score-Based Density Ratio Estimation](https://mlsanity.com/paper/2604.10672/) (2026)
- [InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate](https://mlsanity.com/paper/2606.00241/) (2026)
- [Empirical Risk Minimization with f-Divergence Regularization](https://mlsanity.com/paper/2601.13191/) (2026)
- [Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation](https://mlsanity.com/paper/2607.03487/) (2026)
- [Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Models](https://mlsanity.com/paper/2609.39445/) (2026)

## Abstract (arXiv)

Estimating mutual information accurately is pivotal across diverse applications, from machine learning to communications and biology, enabling us to gain insights into the inner mechanisms of complex systems. Yet, dealing with high-dimensional data presents a formidable challenge, due to its size and the presence of intricate relationships. Recently proposed neural methods employing variational lower bounds on the mutual information have gained prominence. However, these approaches suffer from either high bias or high variance, as the sample size and the structure of the loss function directly influence the training process. In this paper, we propose a novel class of discriminative mutual information estimators based on the variational representation of the f-divergence. We investigate the impact of the permutation function used to obtain the marginal training samples and present a novel architectural solution based on derangements. The proposed estimator is flexible since it exhibits an excellent bias/variance trade-off. The comparison with state-of-the-art neural estimators, through extensive experimentation within established reference scenarios, shows that our approach offers higher accuracy and lower complexity.