# InfoNCE is variational inference in a recognition parameterised model

arXiv: https://arxiv.org/abs/2107.02495 · HTML: https://mlsanity.com/paper/2107.02495/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 6 Jul 2021.

## Builds on (most cited first, 26 of 26)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Variational Lossy Autoencoder](https://mlsanity.com/paper/1611.02731/) (2016)
- [Self-Supervised Learning with Kernel Dependence Maximization](https://mlsanity.com/paper/2106.08320/) (2021)
- [SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows](https://mlsanity.com/paper/2007.02731/) (2020)
- [Capturing Label Characteristics in VAEs](https://mlsanity.com/paper/2006.10102/) (2020)
- [Bayesian representation learning with oracle constraints](https://mlsanity.com/paper/1506.05011/) (2015)

## Built on by (most cited first, 10 of 10)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [An Information Theoretic Perspective on Agentic System Design](https://mlsanity.com/paper/2512.21720/) (2025)
- [Understanding the Role of Equivariance in Self-supervised Learning](https://mlsanity.com/paper/2411.06508/) (2024)
- [Variational Supervised Contrastive Learning](https://mlsanity.com/paper/2506.07413/) (2025)
- [Understanding Self-Supervised Learning via Latent Distribution Matching](https://mlsanity.com/paper/2605.03517/) (2026)
- [Disentangled Representation Learning with the Gromov-Monge Gap](https://mlsanity.com/paper/2407.07829/) (2024)
- [Beyond Spectral Decomposition: Bayesian Contrastive Learning and its Non-negative Formulation via Factor Analysis](https://mlsanity.com/paper/2407.21740/) (2024)
- [Probabilistic Variational Contrastive Learning](https://mlsanity.com/paper/2506.10159/) (2025)
- [Understanding Self-supervised Contrastive Learning through Supervised Objectives](https://mlsanity.com/paper/2510.10572/) (2025)
- [Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisance](https://mlsanity.com/paper/2609.37789/) (2026)

## Abstract (arXiv)

Here, we show that the InfoNCE objective is equivalent to the ELBO in a new class of probabilistic generative model, the recognition parameterised model (RPM). When we learn the optimal prior, the RPM ELBO becomes equal to the mutual information (MI; up to a constant), establishing a connection to pre-existing self-supervised learning methods such as InfoNCE. However, practical InfoNCE methods do not use the MI as an objective; the MI is invariant to arbitrary invertible transformations, so using an MI objective can lead to highly entangled representations (Tschannen et al., 2019). Instead, the actual InfoNCE objective is a simplified lower bound on the MI which is loose even in the infinite sample limit. Thus, an objective that works (i.e. the actual InfoNCE objective) appears to be motivated as a loose bound on an objective that does not work (i.e. the true MI which gives arbitrarily entangled representations). We give an alternative motivation for the actual InfoNCE objective. In particular, we show that in the infinite sample limit, and for a particular choice of prior, the actual InfoNCE objective is equal to the ELBO (up to a constant); and the ELBO is equal to the marginal likelihood with a deterministic recognition model. Thus, we argue that our VAE perspective gives a better motivation for InfoNCE than MI, as the actual InfoNCE objective is only loosely bounded by the MI, but is equal to the ELBO/marginal likelihood (up to a constant).