# Conditional Noise-Contrastive Estimation of Unnormalised Models

arXiv: https://arxiv.org/abs/1806.03664 · HTML: https://mlsanity.com/paper/1806.03664/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2018.

## Built on by (most cited first, 25 of 25)

- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Density Ratio Estimation via Infinitesimal Classification](https://mlsanity.com/paper/2111.11010/) (2021)
- [Decomposed Mutual Information Estimation for Contrastive Representation Learning](https://mlsanity.com/paper/2106.13401/) (2021)
- [On Out-of-distribution Detection with Energy-based Models](https://mlsanity.com/paper/2107.08785/) (2021)
- [Energy Discrepancies: A Score-Independent Loss for Energy-Based Models](https://mlsanity.com/paper/2307.06431/) (2023)
- ["Noisier" Noise Contrastive Eestimation is (Almost) Maximum Likelihood](https://mlsanity.com/paper/2405.16730/) (2024)
- [Identifiable Energy-based Representations: An Application to Estimating Heterogeneous Causal Effects](https://mlsanity.com/paper/2108.03039/) (2021)
- [Gradient-Guided Importance Sampling for Learning Binary Energy-Based Models](https://mlsanity.com/paper/2210.05782/) (2022)
- [Learning Unnormalized Statistical Models via Compositional Optimization](https://mlsanity.com/paper/2306.07485/) (2023)
- [Statistical applications of contrastive learning](https://mlsanity.com/paper/2204.13999/) (2022)
- [Energy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spaces](https://mlsanity.com/paper/2412.01019/) (2024)
- [Active Slices for Sliced Stein Discrepancy](https://mlsanity.com/paper/2102.03159/) (2021)
- [Optimizing the Noise in Self-Supervised Learning: from Importance Sampling to Noise-Contrastive Estimation](https://mlsanity.com/paper/2301.09696/) (2023)
- [Classification Diffusion Models: Revitalizing Density Ratio Estimation](https://mlsanity.com/paper/2402.10095/) (2024)
- [Binary Losses for Density Ratio Estimation](https://mlsanity.com/paper/2407.01371/) (2024)
- [A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation](https://mlsanity.com/paper/2409.18209/) (2024)
- [Learning Energy-Based Models from Stochastic Interpolants using Spatiotemporal Differences](https://mlsanity.com/paper/2605.26850/) (2026)
- [Self-Adapting Noise-Contrastive Estimation for Energy-Based Models](https://mlsanity.com/paper/2211.02650/) (2022)
- [Master's Thesis: Out-of-distribution Detection with Energy-based Models](https://mlsanity.com/paper/2302.12002/) (2023)
- [Fully Variational Noise-Contrastive Estimation](https://mlsanity.com/paper/2304.02473/) (2023)
- [On the connection between Noise-Contrastive Estimation and Contrastive Divergence](https://mlsanity.com/paper/2402.16688/) (2024)
- [And/or trade-off in artificial neurons: impact on adversarial robustness](https://mlsanity.com/paper/2102.07389/) (2021)
- [Training Discrete Energy-Based Models with Energy Discrepancy](https://mlsanity.com/paper/2307.07595/) (2023)
- [A Doeblin-Anchored Contrastive Chart for Learning Markov Transition Kernels](https://mlsanity.com/paper/2606.02232/) (2026)

## Abstract (arXiv)

Many parametric statistical models are not properly normalised and only specified up to an intractable partition function, which renders parameter estimation difficult. Examples of unnormalised models are Gibbs distributions, Markov random fields, and neural network models in unsupervised deep learning. In previous work, the estimation principle called noise-contrastive estimation (NCE) was introduced where unnormalised models are estimated by learning to distinguish between data and auxiliary noise. An open question is how to best choose the auxiliary noise distribution. We here propose a new method that addresses this issue. The proposed method shares with NCE the idea of formulating density estimation as a supervised learning problem but in contrast to NCE, the proposed method leverages the observed data when generating noise samples. The noise can thus be generated in a semi-automated manner. We first present the underlying theory of the new method, show that score matching emerges as a limiting case, validate the method on continuous and discrete valued synthetic data, and show that we can expect an improved performance compared to NCE when the data lie in a lower-dimensional manifold. Then we demonstrate its applicability in unsupervised deep learning by estimating a four-layer neural image model.