mlsanityData to 7 Oct 2026

LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval

arXiv 2203.06169 ↗First posted 11 Mar 2022cs.CL cs.AI cs.IRcited by 24

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose LaPraDoR, a pretrained dual-tower dense retriever that does not require any supervised data for training. Specifically, we first present Iterative Contrastive Learning (ICoL) that iteratively trains the query and document encoders with a cache mechanism. ICoL not only enlarges the number of negative instances but also keeps representations of cached examples in the same hidden space. We then propose Lexicon-Enhanced Dense Retrieval (LEDR) as a simple yet effective way to enhance dense retrieval with lexical matching. We evaluate LaPraDoR on the recently proposed BEIR benchmark, including 18 datasets of 9 zero-shot text retrieval tasks. Experimental results show that LaPraDoR achieves state-of-the-art performance compared with supervised dense retrieval models, and further analysis reveals the effectiveness of our training strategy and objectives. Compared to re-ranking, our lexicon-enhanced approach can be run in milliseconds (22.5x faster) while achieving superior performance.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 44 of its 73 references that are in the map. First posted on arXiv 11 Mar 2022.

Builds on

44 of 73 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
32 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  2. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  8. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  9. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  10. 2019Passage Re-ranking with BERTcited by 629
  11. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  12. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  13. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  14. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  15. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  16. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  17. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 228
  18. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  19. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  20. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  21. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  22. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  23. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 143
  24. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  25. 2019Learning Dense Representations for Entity Retrievalcited by 122
  26. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  27. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  28. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 68
  29. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  30. 2020Efficient Document Re-Ranking for Transformers by Precomputing Term Representationscited by 44
  31. 2019Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsingcited by 18
  32. 2020Evidence-Aware Inferential Text Generation with Vector Quantised Variational AutoEncodercited by 6

Built on by

Cited by year

Citing papers per year: 2022: 12; 2023: 7; 2024: 2; 2025: 3.

24 papers.

Sort:

  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  3. 2022An Efficiency Study for SPLADE Modelscited by 26
  4. 2022A Thorough Examination on Zero-shot Dense Retrievalcited by 21
  5. 2022SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrievalcited by 21
  6. 2022COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learningcited by 19
  7. 2024Contextual Document Embeddingscited by 19
  8. 2022Mixed-modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQAcited by 17
  9. 2022LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrievalcited by 15
  10. 2023BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Languagecited by 15
  11. 2023Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboardcited by 15
  12. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
12 more
  1. 2022MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrieverscited by 7
  2. 2025Prior-Fitted Networks Scale to Larger Datasets When Treated as Weak Learnerscited by 7
  3. 2023I3: Intent-Introspective Retrieval Conditioned on Instructionscited by 5
  4. 2023Unsupervised Pretraining for Fact Verification by Language Model Distillationcited by 5
  5. 2022Disentangled Modeling of Domain and Relevance for Adaptable Dense Retrievalcited by 4
  6. 2025Revela: Dense Retriever Learning via Language Modelingcited by 3
  7. 2022Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gapscited by 2
  8. 2023Generate to Understand for Representationcited by 0
  9. 2023Rule By Example: Harnessing Logical Rules for Explainable Hate Speech Detectioncited by 0
  10. 2023Unsupervised Domain Adaption for Neural Information Retrievalcited by 0
  11. 2024Domain Adaptation of Multilingual Semantic Search -- Literature Reviewcited by 0
  12. 2025Zero-Shot Contextual Embeddings via Offline Synthetic Corpus Generationcited by 0