mlsanityData to 7 Oct 2026

Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency

arXiv 1809.01812 ↗First posted 6 Sep 2018cs.CL cs.LG stat.MEcited by 103

Not on the map. Why?

Abstract (arXiv)

Noise Contrastive Estimation (NCE) is a powerful parameter estimation method for log-linear models, which avoids calculation of the partition function or its derivatives at each training step, a computationally demanding step in many cases. It is closely related to negative sampling methods, now widely used in NLP. This paper considers NCE-based estimation of conditional models. Conditional models are frequently encountered in practice; however there has not been a rigorous theoretical analysis of NCE in this setting, and we will argue there are subtle but important questions when generalizing NCE to the conditional case. In particular, we analyze two variants of NCE for conditional models: one based on a classification objective, the other based on a ranking objective. We show that the ranking-based variant of NCE gives consistent parameter estimates under weaker assumptions than the classification-based method; we analyze the statistical efficiency of the ranking-based and classification-based variants of NCE; finally we describe experiments on synthetic data and language modeling showing the effectiveness and trade-offs of both methods.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 17; 2023: 19; 2024: 17; 2025: 21; 2026: 14.

103 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 248
  2. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  3. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  4. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  5. 2024Energy-Based Diffusion Language Models for Text Generationcited by 86
  6. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  7. 2022HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequencescited by 57
  8. 2025Horizon Reduction Makes RL Scalablecited by 55
  9. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54
  10. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  11. 2021Lossy Compression for Lossless Predictioncited by 51
  12. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
38 more
  1. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  2. 2021MiCE: Mixture of Contrastive Experts for Unsupervised Image Clusteringcited by 42
  3. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  4. 2021Understanding Hard Negatives in Noise Contrastive Estimationcited by 39
  5. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 39
  6. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  7. 2023Contrastive Difference Predictive Codingcited by 35
  8. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  9. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  10. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 29
  11. 2022From t-SNE to UMAP with contrastive learningcited by 28
  12. 2023Logical Message Passing Networks with One-hop Inference on Atomic Formulascited by 28
  13. 2021Decomposed Mutual Information Estimation for Contrastive Representation Learningcited by 24
  14. 2022Contrasting the landscape of contrastive and non-contrastive learningcited by 24
  15. 2021Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCEcited by 22
  16. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 20
  17. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
  18. 2021Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimizationcited by 18
  19. 2024Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learningcited by 18
  20. 2023Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regressioncited by 17
  21. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
  22. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
  23. 2025Episodic Novelty Through Temporal Distancecited by 15
  24. 2021Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Modelscited by 14
  25. 2024MoGU: A Framework for Enhancing Safety of Open-Sourced LLMs While Preserving Their Usabilitycited by 14
  26. 2023Towards a Unified Framework of Contrastive Learning for Disentangled Representationscited by 13
  27. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  28. 2023Explicit Planning Helps Language Models in Logical Reasoningcited by 12
  29. 2022The Optimal Noise in Noise-Contrastive Learning Is Not What You Thinkcited by 11
  30. 2022Fact Checking with Insufficient Evidencecited by 11
  31. 2023All4One: Symbiotic Neighbour Contrastive Learning via Self-Attention and Redundancy Reductioncited by 11
  32. 2024BBox-Adapter: Lightweight Adapting for Black-Box Large Language Modelscited by 11
  33. 2021Residual Energy-Based Models for End-to-End Speech Recognitioncited by 9
  34. 2023Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Modelscited by 9
  35. 2025Towards Better Instruction Following Retrieval Modelscited by 9
  36. 2022Generative Cooperative Networks for Natural Language Generationcited by 8
  37. 2023Language Model Decoding as Direct Metrics Optimizationcited by 8
  38. 2023Time-series Generation by Contrastive Imitationcited by 8

50 most cited of 103 All 103 (JSON)