mlsanityData to 7 Oct 2026

Understanding Hard Negatives in Noise Contrastive Estimation

arXiv 2104.06245 ↗First posted 13 Apr 2021cs.CL cs.LGcited by 39

Not on the map. Why?

Abstract (arXiv)

The choice of negative examples is important in noise contrastive estimation. Recent works find that hard negatives -- highest-scoring incorrect examples under the model -- are effective in practice, but they are used without a formal justification. We develop analytical tools to understand the role of hard negatives. Specifically, we view the contrastive loss as a biased estimator of the gradient of the cross-entropy loss, and show both theoretically and empirically that setting the negative distribution to be the model distribution results in bias reduction. We also derive a general form of the score function that unifies various architectures used in text retrieval. By combining hard negatives with appropriate score functions, we obtain strong results on the challenging task of zero-shot entity linking.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 20 of its 40 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

20 of 40 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  8. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  9. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  10. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  11. 2020Contrastive Learning with Hard Negative Samplescited by 619
  12. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
8 more
  1. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  2. 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 164
  3. 2019Zero-Shot Entity Linking by Reading Entity Descriptionscited by 124
  4. 2019Learning Dense Representations for Entity Retrievalcited by 122
  5. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  6. 2017Deep Joint Entity Disambiguation with Local Neural Attentioncited by 79
  7. 2019Sampled Softmax with Random Fourier Featurescited by 33
  8. 2020Empirical Evaluation of Pretraining Strategies for Supervised Entity Linkingcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 12; 2024: 6; 2025: 4; 2026: 5.

39 papers.

Sort:

By area:

  1. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  2. 2021Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysiscited by 41
  3. 2021EntQA: Entity Linking as Question Answeringcited by 34
  4. 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 33
  5. 2023Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Miningcited by 25
  6. 2021Entity Linking and Discovery via Arborescence-based Supervised Clusteringcited by 22
  7. 2023No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Functioncited by 19
  8. 2024Contextual Document Embeddingscited by 19
  9. 2022Detecting Label Errors by using Pre-Trained Language Modelscited by 16
  10. 2023Composition-contrastive Learning for Sentence Embeddingscited by 14
  11. 2022Energy-Based Contrastive Learning of Visual Representationscited by 13
  12. 2022Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?cited by 13
27 more
  1. 2022Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorizationcited by 10
  2. 2021MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representationscited by 9
  3. 2023Mini-Batch Optimization of Contrastive Losscited by 8
  4. 2024Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?cited by 8
  5. 2025Bridging Legal Knowledge and AI: Retrieval-Augmented Generation with Vector Stores, Knowledge Graphs, and Hierarchical Non-negative Matrix Factorizationcited by 8
  6. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentationcited by 6
  7. 2023A Read-and-Select Framework for Zero-shot Entity Linkingcited by 6
  8. 2022Global Contrastive Batch Sampling via Optimization on Sample Permutationscited by 5
  9. 2024KET-QA: A Dataset for Knowledge Enhanced Table Question Answeringcited by 5
  10. 2021On a Benefit of Mask Language Modeling: Robustness to Simplicity Biascited by 4
  11. 2024Entity Disambiguation via Fusion Entity Decodingcited by 4
  12. 2022Generating Negative Samples for Sequential Recommendationcited by 3
  13. 2023Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Headcited by 3
  14. 2023Towards Better Entity Linking with Multi-View Enhanced Distillationcited by 3
  15. 2023Unified Embedding Based Personalized Retrieval in Etsy Searchcited by 3
  16. 2024Explaining Relationships Among Research Paperscited by 3
  17. 2023Proxy-based Zero-Shot Entity Linking by Effective Candidate Retrievalcited by 2
  18. 2024Comparing Neighbors Together Makes it Easy: Jointly Comparing Multiple Candidates for Efficient and Effective Retrievalcited by 2
  19. 2023A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classificationcited by 1
  20. 2026ReNCE: Learning to Reason by Noise Contrastive Estimationcited by 1
  21. 2022Active entailment encoding for explanation tree construction using parsimonious generation of hard negativescited by 0
  22. 2025Fake & Square: Training Self-Supervised Vision Transformers with Synthetic Data and Synthetic Hard Negativescited by 0
  23. 2025PGMEL: Policy Gradient-based Generative Adversarial Network for Multimodal Entity Linkingcited by 0
  24. 2026CLEAR: Context Augmentation from Contrastive Learning of Experience via Agentic Reflectioncited by 0
  25. 2026Relevance-Based Embeddings: Lightweight Candidate Retrieval via Heavy-Ranker Callscited by 0
  26. 2026Implicit Manipulation for Skill Selection in LLM Agents with Semantic Matchingcited by 0
  27. 2026BELXTR: Biomedical Entity Linking via Contextualized Token Retrievalcited by 0