# Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training

arXiv: https://arxiv.org/abs/2306.03166 · HTML: https://mlsanity.com/paper/2306.03166/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://mlsanity.com/paper/2108.05540/) (2021)
- [RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder](https://mlsanity.com/paper/2205.12035/) (2022)
- [Crafting Better Contrastive Views for Siamese Representation Learning](https://mlsanity.com/paper/2202.03278/) (2022)
- [Learning to Retrieve Passages without Supervision](https://mlsanity.com/paper/2112.07708/) (2021)
- [Pre-trained Language Model for Web-scale Retrieval in Baidu Search](https://mlsanity.com/paper/2106.03373/) (2021)
- [Object-Aware Cropping for Self-Supervised Learning](https://mlsanity.com/paper/2112.00319/) (2021)

## Built on by (most cited first, 30 of 35)

- [LongLaMP: A Benchmark for Personalized Long-form Text Generation](https://mlsanity.com/paper/2407.11016/) (2024)
- [Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestion](https://mlsanity.com/paper/2311.06318/) (2023)
- [Personalized Text Generation with Contrastive Activation Steering](https://mlsanity.com/paper/2503.05213/) (2025)
- [Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generation](https://mlsanity.com/paper/2512.06690/) (2025)
- [Soft Prompt Tuning for Augmenting Dense Retrieval with Large Language Models](https://mlsanity.com/paper/2307.08303/) (2023)
- [CancerLLM: A Large Language Model in Cancer Domain](https://mlsanity.com/paper/2406.10459/) (2024)
- [Personalized Graph-Based Retrieval for Large Language Models](https://mlsanity.com/paper/2501.02157/) (2025)
- [InfoGain-RAG: Boosting Retrieval-Augmented Generation via Document Information Gain-based Reranking and Filtering](https://mlsanity.com/paper/2509.12765/) (2025)
- [COCOA: CBT-based Conversational Counseling Agent using Memory Specialized in Cognitive Distortions and Dynamic Prompt](https://mlsanity.com/paper/2402.17546/) (2024)
- [W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering](https://mlsanity.com/paper/2408.08444/) (2024)
- [An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking](https://mlsanity.com/paper/2411.05508/) (2024)
- [Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism](https://mlsanity.com/paper/2510.13170/) (2025)
- [EmoRAG: Evaluating RAG Robustness to Symbolic Perturbations](https://mlsanity.com/paper/2512.01335/) (2025)
- [Making Large Language Models Efficient Dense Retrievers](https://mlsanity.com/paper/2512.20612/) (2025)
- [SEGMENT+: Long Text Processing with Short-Context Language Models](https://mlsanity.com/paper/2410.06519/) (2024)
- [NILE: Internal Consistency Alignment in Large Language Models](https://mlsanity.com/paper/2412.16686/) (2024)
- [DYCP: Dynamic Context Pruning for Long-Form Dialogue with LLMs](https://mlsanity.com/paper/2601.07994/) (2026)
- [Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings](https://mlsanity.com/paper/2601.11124/) (2026)
- [Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory](https://mlsanity.com/paper/2605.31086/) (2026)
- [IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment](https://mlsanity.com/paper/2606.08016/) (2026)
- [RAGuard: A Layered Defense Framework for Retrieval-Augmented Generation Systems Against Data Poisoning](https://mlsanity.com/paper/2607.26339/) (2026)
- [Oreo: A Plug-in Context Reconstructor to Enhance Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.13019/) (2025)
- [Beyond Prompting: An Efficient Embedding Framework for Open-Domain Question Answering](https://mlsanity.com/paper/2503.01606/) (2025)
- [TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation](https://mlsanity.com/paper/2504.18269/) (2025)
- [Iterative Critique-Refine Framework for Enhancing LLM Personalization](https://mlsanity.com/paper/2510.24469/) (2025)
- [Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editing](https://mlsanity.com/paper/2511.12661/) (2025)
- [Training Dense Retrievers with Multiple Positive Passages](https://mlsanity.com/paper/2602.12727/) (2026)
- [Who Benefits from RAG? The Role of Exposure, Utility and Attribution Bias](https://mlsanity.com/paper/2603.24218/) (2026)
- [PrefReward: Learning User Preference Matrix for Personalized Text Generation](https://mlsanity.com/paper/2607.21067/) (2026)
- [TimelyRAG: Semantic-Temporal Hybrid Retrieval for Time-Critical Question Answering in Overlapping-Evolving Documents](https://mlsanity.com/paper/2609.11572/) (2026)

## Abstract (arXiv)

Dense retrievers have achieved impressive performance, but their demand for abundant training data limits their application scenarios. Contrastive pre-training, which constructs pseudo-positive examples from unlabeled data, has shown great potential to solve this problem. However, the pseudo-positive examples crafted by data augmentations can be irrelevant. To this end, we propose relevance-aware contrastive learning. It takes the intermediate-trained model itself as an imperfect oracle to estimate the relevance of positive pairs and adaptively weighs the contrastive loss of different pairs according to the estimated relevance. Our method consistently improves the SOTA unsupervised Contriever model on the BEIR and open-domain QA retrieval benchmarks. Further exploration shows that our method can not only beat BM25 after further pre-training on the target corpus but also serves as a good few-shot learner. Our code is publicly available at https://github.com/Yibin-Lei/ReContriever.