# Analysis of Noise Contrastive Estimation from the Perspective of Asymptotic Variance

arXiv: https://arxiv.org/abs/1808.07983 · HTML: https://mlsanity.com/paper/1808.07983/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Aug 2018.

## Built on by (most cited first, 6 of 6)

- [Provable benefits of annealing for estimating normalizing constants: Importance Sampling, Noise-Contrastive Estimation, and beyond](https://mlsanity.com/paper/2310.03902/) (2023)
- [The Optimal Noise in Noise-Contrastive Learning Is Not What You Think](https://mlsanity.com/paper/2203.01110/) (2022)
- [Optimizing the Noise in Self-Supervised Learning: from Importance Sampling to Noise-Contrastive Estimation](https://mlsanity.com/paper/2301.09696/) (2023)
- [A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation](https://mlsanity.com/paper/2409.18209/) (2024)
- [Learning Energy-Based Models from Stochastic Interpolants using Spatiotemporal Differences](https://mlsanity.com/paper/2605.26850/) (2026)
- [Adversarially Contrastive Estimation of Conditional Neural Processes](https://mlsanity.com/paper/2303.13004/) (2023)

## Abstract (arXiv)

There are many models, often called unnormalized models, whose normalizing constants are not calculated in closed form. Maximum likelihood estimation is not directly applicable to unnormalized models. Score matching, contrastive divergence method, pseudo-likelihood, Monte Carlo maximum likelihood, and noise contrastive estimation (NCE) are popular methods for estimating parameters of such models. In this paper, we focus on NCE. The estimator derived from NCE is consistent and asymptotically normal because it is an M-estimator. NCE characteristically uses an auxiliary distribution to calculate the normalizing constant in the same spirit of the importance sampling. In addition, there are several candidates as objective functions of NCE. We focus on how to reduce asymptotic variance. First, we propose a method for reducing asymptotic variance by estimating the parameters of the auxiliary distribution. Then, we determine the form of the objective functions, where the asymptotic variance takes the smallest values in the original estimator class and the proposed estimator classes. We further analyze the robustness of the estimator.