# Notes on Noise Contrastive Estimation and Negative Sampling

arXiv: https://arxiv.org/abs/1410.8251 · HTML: https://mlsanity.com/paper/1410.8251/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2014.

## Built on by (most cited first, 26 of 26)

- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [From t-SNE to UMAP with contrastive learning](https://mlsanity.com/paper/2206.01816/) (2022)
- [PARFormer: Transformer-based Multi-Task Network for Pedestrian Attribute Recognition](https://mlsanity.com/paper/2304.07230/) (2023)
- [CLIPose: Category-Level Object Pose Estimation with Pre-trained Vision-Language Knowledge](https://mlsanity.com/paper/2402.15726/) (2024)
- [Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation](https://mlsanity.com/paper/2110.11271/) (2021)
- [Degradation-Aware Self-Attention Based Transformer for Blind Image Super-Resolution](https://mlsanity.com/paper/2310.04180/) (2023)
- [Comprehensive Analysis of Negative Sampling in Knowledge Graph Representation Learning](https://mlsanity.com/paper/2206.10140/) (2022)
- [Residual2Vec: Debiasing graph embedding with random graphs](https://mlsanity.com/paper/2110.07654/) (2021)
- [Knowledge Graph Embeddings in the Biomedical Domain: Are They Useful? A Look at Link Prediction, Rule Learning, and Downstream Polypharmacy Tasks](https://mlsanity.com/paper/2305.19979/) (2023)
- [Exploring the Limits of Few-Shot Link Prediction in Knowledge Graphs](https://mlsanity.com/paper/2102.03419/) (2021)
- [Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embedding](https://mlsanity.com/paper/2106.07250/) (2021)
- [BoltzNCE: Learning Likelihoods for Boltzmann Generation with Stochastic Interpolants and Noise Contrastive Estimation](https://mlsanity.com/paper/2507.00846/) (2025)
- [Calibrating and Improving Graph Contrastive Learning](https://mlsanity.com/paper/2101.11525/) (2021)
- [Intra-agent speech permits zero-shot task acquisition](https://mlsanity.com/paper/2206.03139/) (2022)
- [Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness](https://mlsanity.com/paper/2205.00001/) (2022)
- [Word Embedding for Social Sciences: An Interdisciplinary Survey](https://mlsanity.com/paper/2207.03086/) (2022)
- [EG-ConMix: An Intrusion Detection Method based on Graph Contrastive Learning](https://mlsanity.com/paper/2403.17980/) (2024)
- [On Sampling-Based Training Criteria for Neural Language Modeling](https://mlsanity.com/paper/2104.10507/) (2021)
- [Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation](https://mlsanity.com/paper/2302.01555/) (2023)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [Weather Analogs with a Machine Learning Similarity Metric for Renewable Resource Forecasting](https://mlsanity.com/paper/2103.04530/) (2021)
- [Neural networks for learning personality traits from natural language](https://mlsanity.com/paper/2302.13782/) (2023)
- [Breast tumor classification based on self-supervised contrastive learning from ultrasound videos](https://mlsanity.com/paper/2408.10600/) (2024)
- [Learning Point Cloud Representations with Pose Continuity for Depth-Based Category-Level 6D Object Pose Estimation](https://mlsanity.com/paper/2508.14358/) (2025)
- [VEMamba: Efficient Isotropic Reconstruction of Volume Electron Microscopy with Axial-Lateral Consistent Mamba](https://mlsanity.com/paper/2603.00887/) (2026)

## Abstract (arXiv)

Estimating the parameters of probabilistic models of language such as maxent models and probabilistic neural models is computationally difficult since it involves evaluating partition functions by summing over an entire vocabulary, which may be millions of word types in size. Two closely related strategies---noise contrastive estimation (Mnih and Teh, 2012; Mnih and Kavukcuoglu, 2013; Vaswani et al., 2013) and negative sampling (Mikolov et al., 2012; Goldberg and Levy, 2014)---have emerged as popular solutions to this computational problem, but some confusion remains as to which is more appropriate and when. This document explicates their relationships to each other and to other estimation techniques. The analysis shows that, although they are superficially similar, NCE is a general parameter estimation technique that is asymptotically unbiased, while negative sampling is best understood as a family of binary classification models that are useful for learning word representations but not as a general-purpose estimator.