mlsanityData to 7 Oct 2026

Unsupervised Dense Information Retrieval with Contrastive Learning

arXiv 2112.09118 ↗First posted 16 Dec 2021cs.IR cs.AI cs.CLcited by 1,203

Not on the map. Why?

Cited by 438 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, information retrieval has seen the emergence of dense retrievers, using neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained state-of-the-art results on datasets and tasks where large training sets are available. However, they do not transfer well to new applications with no training data, and are outperformed by unsupervised term-frequency methods such as BM25. In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers and show that it leads to strong performance in various retrieval settings. On the BEIR benchmark our unsupervised model outperforms BM25 on 11 out of 15 datasets for the Recall@100. When used as pre-training before fine-tuning, either on a few thousands in-domain examples or on the large MS~MARCO dataset, our contrastive model leads to improvements on the BEIR benchmark. Finally, we evaluate our approach for multi-lingual retrieval, where training data is even scarcer than for English, and show that our approach leads to strong unsupervised performance. Our model also exhibits strong cross-lingual transfer when fine-tuned on supervised English data only and evaluated on low resources language such as Swahili. We show that our unsupervised models can perform cross-lingual retrieval between different scripts, such as retrieving English documents from Arabic queries, which would not be possible with term matching methods.

Key facts. Cited by 1,203 papers in the map (data through 7 Oct 2026). Builds on 53 of its 68 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

53 of 68 references.

References · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2017Billion-scale similarity search with GPUscited by 2,477
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
38 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  2. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  6. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  7. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  8. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  9. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  10. 2019Passage Re-ranking with BERTcited by 629
  11. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  12. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  13. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  14. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  15. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  16. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  17. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  18. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  19. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  20. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  21. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  22. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  23. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  24. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  25. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  26. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  27. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  28. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  29. 2020Pre-training via Paraphrasingcited by 92
  30. 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answeringcited by 75
  31. 2017On Sampling Strategies for Neural Network-based Collaborative Filteringcited by 71
  32. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  33. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 59
  34. 2015Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrievalcited by 54
  35. 2018End-to-End Retrieval in Continuous Spacecited by 46
  36. 2021One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrievalcited by 46
  37. 2021Learning to Retrieve Passages without Supervisioncited by 46
  38. 2021A Replication Study of Dense Passage Retrievercited by 37

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 35; 2023: 138; 2024: 362; 2025: 364; 2026: 304.

1,203 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  6. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  8. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  9. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  10. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  11. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  12. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
38 more
  1. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  2. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  3. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  4. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  5. 2023Cognitive Architectures for Language Agentscited by 422
  6. 2024The Faiss librarycited by 370
  7. 2023Adapting Language Models to Compress Contextscited by 350
  8. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  9. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  10. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  11. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  12. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  13. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
  14. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  15. 2022Prompting GPT-3 To Be Reliablecited by 258
  16. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  17. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  18. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  19. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  20. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  21. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  22. 2024Tool Learning with Large Language Models: A Surveycited by 221
  23. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  24. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  25. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  26. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  27. 2024Corrective Retrieval Augmented Generationcited by 187
  28. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  29. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  30. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  31. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  32. 2023Representation Learning with Large Language Models for Recommendationcited by 155
  33. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
  34. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  35. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
  36. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  37. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 146
  38. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145

50 most cited of 1,203 All 1,203 (JSON)