# Understanding Hard Negatives in Noise Contrastive Estimation

arXiv: https://arxiv.org/abs/2104.06245 · HTML: https://mlsanity.com/paper/2104.06245/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020)
- [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019)
- [Zero-Shot Entity Linking by Reading Entity Descriptions](https://mlsanity.com/paper/1906.07348/) (2019)
- [Learning Dense Representations for Entity Retrieval](https://mlsanity.com/paper/1909.10506/) (2019)
- [Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency](https://mlsanity.com/paper/1809.01812/) (2018)
- [Deep Joint Entity Disambiguation with Local Neural Attention](https://mlsanity.com/paper/1704.04920/) (2017)
- [Sampled Softmax with Random Fourier Features](https://mlsanity.com/paper/1907.10747/) (2019)
- [Empirical Evaluation of Pretraining Strategies for Supervised Entity Linking](https://mlsanity.com/paper/2005.14253/) (2020)

## Built on by (most cited first, 30 of 39)

- [Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training](https://mlsanity.com/paper/2301.02280/) (2023)
- [Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.01797/) (2021)
- [Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning](https://mlsanity.com/paper/2505.12477/) (2025)
- [EntQA: Entity Linking as Question Answering](https://mlsanity.com/paper/2110.02369/) (2021)
- [Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining](https://mlsanity.com/paper/2311.03964/) (2023)
- [Entity Linking and Discovery via Arborescence-based Supervised Clustering](https://mlsanity.com/paper/2109.01242/) (2021)
- [No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Function](https://mlsanity.com/paper/2309.03224/) (2023)
- [Contextual Document Embeddings](https://mlsanity.com/paper/2410.02525/) (2024)
- [Detecting Label Errors by using Pre-Trained Language Models](https://mlsanity.com/paper/2205.12702/) (2022)
- [Composition-contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2307.07380/) (2023)
- [Energy-Based Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2202.04933/) (2022)
- [Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?](https://mlsanity.com/paper/2212.10784/) (2022)
- [Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorization](https://mlsanity.com/paper/2210.12579/) (2022)
- [MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations](https://mlsanity.com/paper/2109.05716/) (2021)
- [Mini-Batch Optimization of Contrastive Loss](https://mlsanity.com/paper/2307.05906/) (2023)
- [Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?](https://mlsanity.com/paper/2406.10743/) (2024)
- [Bridging Legal Knowledge and AI: Retrieval-Augmented Generation with Vector Stores, Knowledge Graphs, and Hierarchical Non-negative Matrix Factorization](https://mlsanity.com/paper/2502.20364/) (2025)
- [ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation](https://mlsanity.com/paper/2308.16689/) (2023)
- [A Read-and-Select Framework for Zero-shot Entity Linking](https://mlsanity.com/paper/2310.12450/) (2023)
- [Global Contrastive Batch Sampling via Optimization on Sample Permutations](https://mlsanity.com/paper/2210.12874/) (2022)
- [KET-QA: A Dataset for Knowledge Enhanced Table Question Answering](https://mlsanity.com/paper/2405.08099/) (2024)
- [On a Benefit of Mask Language Modeling: Robustness to Simplicity Bias](https://mlsanity.com/paper/2110.05301/) (2021)
- [Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Head](https://mlsanity.com/paper/2302.10260/) (2023)
- [Unified Embedding Based Personalized Retrieval in Etsy Search](https://mlsanity.com/paper/2306.04833/) (2023)
- [Entity Disambiguation via Fusion Entity Decoding](https://mlsanity.com/paper/2404.01626/) (2024)
- [Generating Negative Samples for Sequential Recommendation](https://mlsanity.com/paper/2208.03645/) (2022)
- [Towards Better Entity Linking with Multi-View Enhanced Distillation](https://mlsanity.com/paper/2305.17371/) (2023)
- [Explaining Relationships Among Research Papers](https://mlsanity.com/paper/2402.13426/) (2024)
- [Proxy-based Zero-Shot Entity Linking by Effective Candidate Retrieval](https://mlsanity.com/paper/2301.13318/) (2023)
- [Comparing Neighbors Together Makes it Easy: Jointly Comparing Multiple Candidates for Efficient and Effective Retrieval](https://mlsanity.com/paper/2405.12801/) (2024)

## Abstract (arXiv)

The choice of negative examples is important in noise contrastive estimation. Recent works find that hard negatives -- highest-scoring incorrect examples under the model -- are effective in practice, but they are used without a formal justification. We develop analytical tools to understand the role of hard negatives. Specifically, we view the contrastive loss as a biased estimator of the gradient of the cross-entropy loss, and show both theoretically and empirically that setting the negative distribution to be the model distribution results in bias reduction. We also derive a general form of the score function that unifies various architectures used in text retrieval. By combining hard negatives with appropriate score functions, we obtain strong results on the challenging task of zero-shot entity linking.