# On the connection between Noise-Contrastive Estimation and Contrastive Divergence

arXiv: https://arxiv.org/abs/2402.16688 · HTML: https://mlsanity.com/paper/2402.16688/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 53 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Exploring the Limits of Language Modeling](https://mlsanity.com/paper/1602.02410/) (2016)
- [Improved Contrastive Divergence Training of Energy Based Models](https://mlsanity.com/paper/2012.01316/) (2020)
- [Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency](https://mlsanity.com/paper/1809.01812/) (2018)
- [Flow Contrastive Estimation of Energy-Based Models](https://mlsanity.com/paper/1912.00589/) (2019)
- [Elements of Sequential Monte Carlo](https://mlsanity.com/paper/1903.04797/) (2019)
- [Markovian Score Climbing: Variational Inference with KL(p||q)](https://mlsanity.com/paper/2003.10374/) (2020)
- [Conditional Noise-Contrastive Estimation of Unnormalised Models](https://mlsanity.com/paper/1806.03664/) (2018)
- [Autoregressive Energy Machines](https://mlsanity.com/paper/1904.05626/) (2019)
- [How to Train Your Energy-Based Model for Regression](https://mlsanity.com/paper/2005.01698/) (2020)
- [Arbitrary Conditional Distributions with Energy](https://mlsanity.com/paper/2102.04426/) (2021)
- [On distinguishability criteria for estimating generative models](https://mlsanity.com/paper/1412.6515/) (2014)
- [The Optimal Noise in Noise-Contrastive Learning Is Not What You Think](https://mlsanity.com/paper/2203.01110/) (2022)
- [Contrastive Divergence Learning is a Time Reversal Adversarial Game](https://mlsanity.com/paper/2012.03295/) (2020)
- [Learning Proposals for Practical Energy-Based Regression](https://mlsanity.com/paper/2110.11948/) (2021)
- [Self-Adapting Noise-Contrastive Estimation for Energy-Based Models](https://mlsanity.com/paper/2211.02650/) (2022)

## Built on by (most cited first, 1 of 1)

- [SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2512.07175/) (2025)

## Abstract (arXiv)

Noise-contrastive estimation (NCE) is a popular method for estimating unnormalised probabilistic models, such as energy-based models, which are effective for modelling complex data distributions. Unlike classical maximum likelihood (ML) estimation that relies on importance sampling (resulting in ML-IS) or MCMC (resulting in contrastive divergence, CD), NCE uses a proxy criterion to avoid the need for evaluating an often intractable normalisation constant. Despite apparent conceptual differences, we show that two NCE criteria, ranking NCE (RNCE) and conditional NCE (CNCE), can be viewed as ML estimation methods. Specifically, RNCE is equivalent to ML estimation combined with conditional importance sampling, and both RNCE and CNCE are special cases of CD. These findings bridge the gap between the two method classes and allow us to apply techniques from the ML-IS and CD literature to NCE, offering several advantageous extensions.