# Multi-label Contrastive Predictive Coding

arXiv: https://arxiv.org/abs/2007.09852 · HTML: https://mlsanity.com/paper/2007.09852/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2020.

## Built on by (most cited first, 30 of 32)

- [Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations](https://mlsanity.com/paper/2203.01517/) (2022)
- [D2C: Diffusion-Denoising Models for Few-shot Conditional Generation](https://mlsanity.com/paper/2106.06819/) (2021)
- [Lossy Compression for Lossless Prediction](https://mlsanity.com/paper/2106.10800/) (2021)
- [Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation](https://mlsanity.com/paper/2206.07771/) (2022)
- [Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction](https://mlsanity.com/paper/2102.10106/) (2021)
- [Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining](https://mlsanity.com/paper/2405.20462/) (2024)
- [Decomposed Mutual Information Estimation for Contrastive Representation Learning](https://mlsanity.com/paper/2106.13401/) (2021)
- [Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithm](https://mlsanity.com/paper/2312.12668/) (2023)
- [Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions](https://mlsanity.com/paper/2111.14629/) (2021)
- [Heterogeneous Contrastive Learning](https://mlsanity.com/paper/2105.09401/) (2021)
- [Gaussian Mixture Variational Autoencoder with Contrastive Learning for Multi-Label Classification](https://mlsanity.com/paper/2112.00976/) (2021)
- [Contrastive Attraction and Contrastive Repulsion for Representation Learning](https://mlsanity.com/paper/2105.03746/) (2021)
- [A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets](https://mlsanity.com/paper/2410.10924/) (2024)
- [Poly-View Contrastive Learning](https://mlsanity.com/paper/2403.05490/) (2024)
- [LeNSE: Learning To Navigate Subgraph Embeddings for Large-Scale Combinatorial Optimisation](https://mlsanity.com/paper/2205.10106/) (2022)
- [RenyiCL: Contrastive Representation Learning with Skew Renyi Divergence](https://mlsanity.com/paper/2208.06270/) (2022)
- [A Low Rank Promoting Prior for Unsupervised Contrastive Learning](https://mlsanity.com/paper/2108.02696/) (2021)
- [Deeper-GXX: Deepening Arbitrary GNNs](https://mlsanity.com/paper/2110.13798/) (2021)
- [Contrastive Learning of Global-Local Video Representations](https://mlsanity.com/paper/2104.05418/) (2021)
- [Integrating Auxiliary Information in Self-supervised Learning](https://mlsanity.com/paper/2106.02869/) (2021)
- [Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration](https://mlsanity.com/paper/2112.08132/) (2021)
- [Learning Weakly-Supervised Contrastive Representations](https://mlsanity.com/paper/2202.06670/) (2022)
- [Forget-me-not! Contrastive Critics for Mitigating Posterior Collapse](https://mlsanity.com/paper/2207.09535/) (2022)
- [Generative prediction of flow fields around an obstacle using the diffusion model](https://mlsanity.com/paper/2407.00735/) (2024)
- [On the Utility of Foundation Models for Fast MRI: Vision-Language-Guided Image Reconstruction](https://mlsanity.com/paper/2511.19641/) (2025)
- [Online Symbolic Regression with Informative Query](https://mlsanity.com/paper/2302.10539/) (2023)
- [Towards a Rigorous Analysis of Mutual Information in Contrastive Learning](https://mlsanity.com/paper/2308.15704/) (2023)
- [Contrastive Predictive Coding Done Right for Mutual Information Estimation](https://mlsanity.com/paper/2510.25983/) (2025)
- [ScoreCL: Augmentation-Adaptive Contrastive Learning via Score-Matching Function](https://mlsanity.com/paper/2306.04175/) (2023)
- [Improving the Consistency in Cross-Lingual Cross-Modal Retrieval with 1-to-K Contrastive Learning](https://mlsanity.com/paper/2406.18254/) (2024)

## Abstract (arXiv)

Variational mutual information (MI) estimators are widely used in unsupervised representation learning methods such as contrastive predictive coding (CPC). A lower bound on MI can be obtained from a multi-class classification problem, where a critic attempts to distinguish a positive sample drawn from the underlying joint distribution from (m-1) negative samples drawn from a suitable proposal distribution. Using this approach, MI estimates are bounded above by log m, and could thus severely underestimate unless m is very large. To overcome this limitation, we introduce a novel estimator based on a multi-label classification problem, where the critic needs to jointly identify multiple positive samples at the same time. We show that using the same amount of negative samples, multi-label CPC is able to exceed the log m bound, while still being a valid lower bound of mutual information. We demonstrate that the proposed approach is able to lead to better mutual information estimation, gain empirical improvements in unsupervised representation learning, and beat a current state-of-the-art knowledge distillation method over 10 out of 13 tasks.