mlsanityData to 14 Oct 2026

Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation

arXiv 2110.11271 ↗First posted 21 Oct 2021cs.LG stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

Noise-contrastive estimation (NCE) is a statistically consistent method for learning unnormalized probabilistic models. It has been empirically observed that the choice of the noise distribution is crucial for NCE's performance. However, such observations have never been made formal or quantitative. In fact, it is not even clear whether the difficulties arising from a poorly chosen noise distribution are statistical or algorithmic in nature. In this work, we formally pinpoint reasons for NCE's poor performance when an inappropriate noise distribution is used. Namely, we prove these challenges arise due to an ill-behaved (more precisely, flat) loss landscape. To address this, we introduce a variant of NCE called "eNCE" which uses an exponential loss and for which normalized gradient descent addresses the landscape issues provably when the target and noise distributions are in a given exponential family.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 19 of its 38 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

19 of 38 references.

References · 19

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2019Contrastive Multiview Codingcited by 1,301
  3. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  4. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  5. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  6. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
  7. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 273
  8. 2020Telescoping Density-Ratio Estimationcited by 108
  9. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  10. 2015Beyond Convexity: Stochastic Quasi-Convex Optimizationcited by 76
  11. 2019Flow Contrastive Estimation of Energy-Based Modelscited by 75
  12. 2014Logistic Regression: Tight Bounds for Stochastic and Online Optimizationcited by 31
7 more
  1. 2020Training Deep Energy-Based Models with f-Divergence Minimizationcited by 29
  2. 2019Simple and optimal high-probability bounds for strongly-convex stochastic gradient descentcited by 27
  3. 2014Notes on Noise Contrastive Estimation and Negative Samplingcited by 26
  4. 2014Accurate and Conservative Estimates of MRF Log-likelihood using Reverse Annealingcited by 18
  5. 2021Heavy-tailed Streaming Statistical Estimationcited by 10
  6. 2018Generative Ratio Matching Networkscited by 6
  7. 2015On the accuracy of self-normalized log-linear modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 4; 2025: 2; 2026: 1.

15 papers.

Sort:

By area:

  1. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 40
  2. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  3. 2023Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regressioncited by 17
  4. 2023Provable benefits of annealing for estimating normalizing constants: Importance Sampling, Noise-Contrastive Estimation, and beyondcited by 13
  5. 2025Density Ratio Estimation with Conditional Probability Pathscited by 13
  6. 2023Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Modelscited by 10
  7. 2024"Noisier" Noise Contrastive Eestimation is (Almost) Maximum Likelihoodcited by 8
  8. 2025BoltzNCE: Learning Likelihoods for Boltzmann Generation with Stochastic Interpolants and Noise Contrastive Estimationcited by 7
  9. 2022Pitfalls of Gaussians as a noise distribution in NCEcited by 6
  10. 2023Learning Unnormalized Statistical Models via Compositional Optimizationcited by 6
  11. 2022Statistical applications of contrastive learningcited by 5
  12. 2023Optimizing the Noise in Self-Supervised Learning: from Importance Sampling to Noise-Contrastive Estimationcited by 2
3 more
  1. 2024A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimationcited by 2
  2. 2024On Discriminative Probabilistic Modeling for Self-Supervised Representation Learningcited by 2
  3. 2026A Geometry-Aware Efficient Algorithm for Compositional Entropic Risk Minimizationcited by 1