# Pitfalls of Gaussians as a noise distribution in NCE

arXiv: https://arxiv.org/abs/2210.00189 · HTML: https://mlsanity.com/paper/2210.00189/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 1 Oct 2022.

## Builds on (most cited first, 6 of 6)

- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [Flow Contrastive Estimation of Energy-Based Models](https://mlsanity.com/paper/1912.00589/) (2019)
- [Implicit Generation and Generalization in Energy-Based Models](https://mlsanity.com/paper/1903.08689/) (2019)
- [Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation](https://mlsanity.com/paper/2110.11271/) (2021)
- [The Optimal Noise in Noise-Contrastive Learning Is Not What You Think](https://mlsanity.com/paper/2203.01110/) (2022)

## Built on by (most cited first, 6 of 6)

- [Provable benefits of annealing for estimating normalizing constants: Importance Sampling, Noise-Contrastive Estimation, and beyond](https://mlsanity.com/paper/2310.03902/) (2023)
- [Density Ratio Estimation with Conditional Probability Paths](https://mlsanity.com/paper/2502.02300/) (2025)
- [Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models](https://mlsanity.com/paper/2309.05803/) (2023)
- [BoltzNCE: Learning Likelihoods for Boltzmann Generation with Stochastic Interpolants and Noise Contrastive Estimation](https://mlsanity.com/paper/2507.00846/) (2025)
- [Optimizing the Noise in Self-Supervised Learning: from Importance Sampling to Noise-Contrastive Estimation](https://mlsanity.com/paper/2301.09696/) (2023)
- [A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation](https://mlsanity.com/paper/2409.18209/) (2024)

## Abstract (arXiv)

Noise Contrastive Estimation (NCE) is a popular approach for learning probability density functions parameterized up to a constant of proportionality. The main idea is to design a classification problem for distinguishing training data from samples from an easy-to-sample noise distribution q, in a manner that avoids having to calculate a partition function. It is well-known that the choice of q can severely impact the computational and statistical efficiency of NCE. In practice, a common choice for q is a Gaussian which matches the mean and covariance of the data. In this paper, we show that such a choice can result in an exponentially bad (in the ambient dimension) conditioning of the Hessian of the loss, even for very simple data distributions. As a consequence, both the statistical and algorithmic complexity for such a choice of q will be problematic in practice, suggesting that more complex noise distributions are essential to the success of NCE.