mlsanityData to 7 Oct 2026

Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval

arXiv 2108.05540 ↗First posted 12 Aug 2021cs.IR cs.CLcited by 203

Not on the map. Why?

Abstract (arXiv)

Recent research demonstrates the effectiveness of using fine-tuned language models~(LM) for dense retrieval. However, dense retrievers are hard to train, typically requiring heavily engineered fine-tuning pipelines to realize their full potential. In this paper, we identify and address two underlying problems of dense retrievers: i)~fragility to training data noise and ii)~requiring large batches to robustly learn the embedding space. We use the recently proposed Condenser pre-training architecture, which learns to condense information into the dense vector through LM pre-training. On top of it, we propose coCondenser, which adds an unsupervised corpus-level contrastive loss to warm up the passage embedding space. Retrieval experiments on MS-MARCO, Natural Question, and Trivia QA datasets show that coCondenser removes the need for heavy data engineering such as augmentation, synthesis, or filtering, as well as the need for large batch training. It shows comparable performance to RocketQA, a state-of-the-art, heavily engineered system, using simple small batch fine-tuning.

Key facts. Cited by 203 papers in the map (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 12 Aug 2021.

Builds on

28 of 37 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2017Billion-scale similarity search with GPUscited by 2,477
16 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  2. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  3. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  4. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  5. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  6. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  7. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
  8. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  9. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  10. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  11. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  12. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  13. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  14. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  15. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  16. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 58; 2023: 54; 2024: 41; 2025: 35; 2026: 7.

203 papers.

Sort:

By area:

  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  3. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  4. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  5. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  6. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  7. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  8. 2024Tool Learning with Large Language Models: A Surveycited by 221
  9. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  10. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 146
  11. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  12. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
38 more
  1. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  2. 2023Dense X Retrieval: What Retrieval Granularity Should We Use?cited by 88
  3. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  4. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 77
  5. 2023Context-faithful Prompting for Large Language Modelscited by 76
  6. 2024SPLADE-v3: New baselines for SPLADEcited by 69
  7. 2023SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrievalcited by 61
  8. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
  9. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  10. 2022Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generationcited by 47
  11. 2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
  12. 2021Learning to Retrieve Passages without Supervisioncited by 46
  13. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  14. 2023Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Rankingcited by 44
  15. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  16. 2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
  17. 2022Curriculum Learning for Dense Retrieval Distillationcited by 38
  18. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  19. 2022Towards Robust Ranker for Text Retrievalcited by 37
  20. 2023LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Datasetcited by 37
  21. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  22. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  23. 2023Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Trainingcited by 35
  24. 2021Open Domain Question Answering with A Unified Knowledge Interfacecited by 32
  25. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  26. 2025Topic-FlipRAG: Topic-Orientated Adversarial Opinion Manipulation Attacks to Retrieval-Augmented Generation Modelscited by 32
  27. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  28. 2022Fine-Grained Distillation for Long Document Retrievalcited by 29
  29. 2023Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Methodcited by 29
  30. 2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28
  31. 2023NevIR: Negation in Neural Information Retrievalcited by 28
  32. 2024Scaling Laws For Dense Retrievalcited by 28
  33. 2022Multi-CPR: A Multi Domain Chinese Dataset for Passage Retrievalcited by 26
  34. 2022An Efficiency Study for SPLADE Modelscited by 26
  35. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  36. 2024Towards Completeness-Oriented Tool Retrieval for Large Language Modelscited by 26
  37. 2022LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrievalcited by 24
  38. 2023Decomposing Complex Queries for Tip-of-the-tongue Retrievalcited by 24

50 most cited of 203 All 203 (JSON)