# Contrastive Predictive Coding Done Right for Mutual Information Estimation

arXiv: https://arxiv.org/abs/2510.25983 · HTML: https://mlsanity.com/paper/2510.25983/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 28 of 28)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning](https://mlsanity.com/paper/2108.01775/) (2021)
- [Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency](https://mlsanity.com/paper/1809.01812/) (2018)
- [On Mutual Information in Contrastive Learning for Visual Representations](https://mlsanity.com/paper/2005.13149/) (2020)
- [Multi-label Contrastive Predictive Coding](https://mlsanity.com/paper/2007.09852/) (2020)
- [Neural Methods for Point-wise Dependency Estimation](https://mlsanity.com/paper/2006.05553/) (2020)
- [An Efficient Approach to Informative Feature Extraction from Multimodal Data](https://mlsanity.com/paper/1811.08979/) (2018)
- [Approximating mutual information of high-dimensional variables using learned representations](https://mlsanity.com/paper/2409.02732/) (2024)
- [Mutual Information Estimation via f-Divergence and Data Derangements](https://mlsanity.com/paper/2305.20025/) (2023)
- [Operator SVD with Neural Networks via Nested Low-Rank Approximation](https://mlsanity.com/paper/2402.03655/) (2024)
- [f-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning](https://mlsanity.com/paper/2402.10150/) (2024)
- [A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets](https://mlsanity.com/paper/2410.10924/) (2024)
- [RenyiCL: Contrastive Representation Learning with Skew Renyi Divergence](https://mlsanity.com/paper/2208.06270/) (2022)
- [Unconstrained Stochastic CCA: Unifying Multiview and Self-Supervised Learning](https://mlsanity.com/paper/2310.01012/) (2023)
- [A Unified Framework for Multi-distribution Density Ratio Estimation](https://mlsanity.com/paper/2112.03440/) (2021)
- [A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation](https://mlsanity.com/paper/2409.18209/) (2024)

## Built on by (most cited first, 2 of 2)

- [Statistical Consistency and Generalization of Contrastive Representation Learning](https://mlsanity.com/paper/2605.02116/) (2026)
- [When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE](https://mlsanity.com/paper/2606.00262/) (2026)

## Abstract (arXiv)

The InfoNCE objective, originally introduced for contrastive representation learning, has become a popular choice for mutual information (MI) estimation, despite its indirect connection to MI. In this paper, we demonstrate why InfoNCE should not be regarded as a valid MI estimator, and we introduce a simple modification, which we refer to as InfoNCE-anchor, for accurate MI estimation. Our modification introduces an auxiliary anchor class, enabling consistent density ratio estimation and yielding a plug-in MI estimator with significantly reduced bias. Beyond this, we generalize our framework using proper scoring rules, which recover InfoNCE-anchor as a special case when the log score is employed. This formulation unifies a broad spectrum of contrastive objectives, including NCE, InfoNCE, and f-divergence variants, under a single principled framework. Empirically, we find that InfoNCE-anchor with the log score achieves the most accurate MI estimates; however, in self-supervised representation learning experiments, we find that the anchor does not improve the downstream task performance. These findings corroborate that contrastive representation learning benefits not from accurate MI estimation per se, but from the learning of structured density ratios.