mlsanityData to 7 Oct 2026

Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training

arXiv 2306.03166 ↗First posted 5 Jun 2023cs.IR cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Dense retrievers have achieved impressive performance, but their demand for abundant training data limits their application scenarios. Contrastive pre-training, which constructs pseudo-positive examples from unlabeled data, has shown great potential to solve this problem. However, the pseudo-positive examples crafted by data augmentations can be irrelevant. To this end, we propose relevance-aware contrastive learning. It takes the intermediate-trained model itself as an imperfect oracle to estimate the relevance of positive pairs and adaptively weighs the contrastive loss of different pairs according to the estimated relevance. Our method consistently improves the SOTA unsupervised Contriever model on the BEIR and open-domain QA retrieval benchmarks. Further exploration shows that our method can not only beat BM25 after further pre-training on the target corpus but also serves as a good few-shot learner. Our code is publicly available at https://github.com/Yibin-Lei/ReContriever.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

20 of 32 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  10. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  11. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  12. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
8 more
  1. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  2. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  3. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  4. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  5. 2022Crafting Better Contrastive Views for Siamese Representation Learningcited by 67
  6. 2021Learning to Retrieve Passages without Supervisioncited by 46
  7. 2021Pre-trained Language Model for Web-scale Retrieval in Baidu Searchcited by 31
  8. 2021Object-Aware Cropping for Self-Supervised Learningcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 13; 2026: 13.

35 papers.

Sort:

By area:

  1. 2024LongLaMP: A Benchmark for Personalized Long-form Text Generationcited by 91
  2. 2023Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestioncited by 43
  3. 2025Personalized Text Generation with Contrastive Activation Steeringcited by 30
  4. 2025Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generationcited by 29
  5. 2023Soft Prompt Tuning for Augmenting Dense Retrieval with Large Language Modelscited by 12
  6. 2024CancerLLM: A Large Language Model in Cancer Domaincited by 11
  7. 2025Personalized Graph-Based Retrieval for Large Language Modelscited by 10
  8. 2025InfoGain-RAG: Boosting Retrieval-Augmented Generation via Document Information Gain-based Reranking and Filteringcited by 8
  9. 2024COCOA: CBT-based Conversational Counseling Agent using Memory Specialized in Cognitive Distortions and Dynamic Promptcited by 7
  10. 2024W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answeringcited by 6
  11. 2024An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Rerankingcited by 4
  12. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
23 more
  1. 2025EmoRAG: Evaluating RAG Robustness to Symbolic Perturbationscited by 3
  2. 2024SEGMENT+: Long Text Processing with Short-Context Language Modelscited by 2
  3. 2024NILE: Internal Consistency Alignment in Large Language Modelscited by 2
  4. 2025Making Large Language Models Efficient Dense Retrieverscited by 2
  5. 2026DYCP: Dynamic Context Pruning for Long-Form Dialogue with LLMscited by 2
  6. 2026Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddingscited by 2
  7. 2026IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignmentcited by 2
  8. 2026RAGuard: A Layered Defense Framework for Retrieval-Augmented Generation Systems Against Data Poisoningcited by 2
  9. 2025Oreo: A Plug-in Context Reconstructor to Enhance Retrieval-Augmented Generationcited by 1
  10. 2025Beyond Prompting: An Efficient Embedding Framework for Open-Domain Question Answeringcited by 1
  11. 2025TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generationcited by 1
  12. 2025Iterative Critique-Refine Framework for Enhancing LLM Personalizationcited by 1
  13. 2025Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editingcited by 1
  14. 2026Training Dense Retrievers with Multiple Positive Passagescited by 1
  15. 2026Who Benefits from RAG? The Role of Exposure, Utility and Attribution Biascited by 1
  16. 2026Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memorycited by 1
  17. 2026TimelyRAG: Semantic-Temporal Hybrid Retrieval for Time-Critical Question Answering in Overlapping-Evolving Documentscited by 1
  18. 2025Retrieval and Augmentation of Domain Knowledge for Text-to-SQL Semantic Parsingcited by 0
  19. 2026ChunQiuTR: Time-Keyed Temporal Retrieval in Classical Chinese Annalscited by 0
  20. 2026BioHiCL: Hierarchical Multi-Label Contrastive Learning for Biomedical Retrieval with MeSH Labelscited by 0
  21. 2026Learning to Route Queries to Heads for Attention-based Re-ranking with Large Language Modelscited by 0
  22. 2026Mask-to-Correct^+: Leveraging Retriever Diversity for Masking-guided Faithful Fact Correctioncited by 0
  23. 2026PrefReward: Learning User Preference Matrix for Personalized Text Generationcited by 0