# COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning

arXiv: https://arxiv.org/abs/2210.15212 · HTML: https://mlsanity.com/paper/2210.15212/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 76 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 30 of 48)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems](https://mlsanity.com/paper/2011.12945/) (2020)

## Built on by (most cited first, 20 of 20)

- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://mlsanity.com/paper/2212.10496/) (2022)
- [ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval](https://mlsanity.com/paper/2305.10703/) (2023)
- [Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaboration](https://mlsanity.com/paper/2504.04915/) (2025)
- [Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence](https://mlsanity.com/paper/2503.05037/) (2025)
- [Synergistic Interplay between Search and Large Language Models for Information Retrieval](https://mlsanity.com/paper/2305.07402/) (2023)
- [Chain-of-Factors Paper-Reviewer Matching](https://mlsanity.com/paper/2310.14483/) (2023)
- [Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training](https://mlsanity.com/paper/2306.07193/) (2023)
- [Beyond Borders: Investigating Cross-Jurisdiction Transfer in Legal Case Summarization](https://mlsanity.com/paper/2403.19317/) (2024)
- [I3: Intent-Introspective Retrieval Conditioned on Instructions](https://mlsanity.com/paper/2308.10025/) (2023)
- [Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval](https://mlsanity.com/paper/2311.15564/) (2023)
- [Large Language Model Augmented Exercise Retrieval for Personalized Language Learning](https://mlsanity.com/paper/2402.16877/) (2024)
- [Investigating Mixture of Experts in Dense Retrieval](https://mlsanity.com/paper/2412.11864/) (2024)
- [PairDistill: Pairwise Relevance Distillation for Dense Retrieval](https://mlsanity.com/paper/2410.01383/) (2024)
- [DREditor: An Time-efficient Approach for Building a Domain-specific Dense Retrieval Model](https://mlsanity.com/paper/2401.12540/) (2024)
- [A Representation Sharpening Framework for Zero Shot Dense Retrieval](https://mlsanity.com/paper/2511.05684/) (2025)
- [With Argus Eyes: Assessing Retrieval Gaps via Uncertainty Scoring to Detect and Remedy Retrieval Blind Spots](https://mlsanity.com/paper/2602.09616/) (2026)
- [Beyond Paper-to-Paper: Structured Profiling and Rubric Scoring for Paper-Reviewer Matching](https://mlsanity.com/paper/2604.05866/) (2026)
- [The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems](https://mlsanity.com/paper/2605.23024/) (2026)
- [CoHyDE: Iterative Co-Training of LLM Rewriter & Dense Encoder for Tool Retrieval](https://mlsanity.com/paper/2605.29271/) (2026)
- [When Does Dense Retrieval Need Asymmetric Geometry? A Bias-Variance Theory of Shared and Dual Projections](https://mlsanity.com/paper/2609.32488/) (2026)

## Abstract (arXiv)

We present a new zero-shot dense retrieval (ZeroDR) method, COCO-DR, to improve the generalization ability of dense retrieval by combating the distribution shifts between source training tasks and target scenarios. To mitigate the impact of document differences, COCO-DR continues pretraining the language model on the target corpora to adapt the model to target distributions via COtinuous COtrastive learning. To prepare for unseen target queries, COCO-DR leverages implicit Distributionally Robust Optimization (iDRO) to reweight samples from different source query clusters for improving model robustness over rare queries during fine-tuning. COCO-DR achieves superior average performance on BEIR, the zero-shot retrieval benchmark. At BERT Base scale, COCO-DR Base outperforms other ZeroDR models with 60x larger size. At BERT Large scale, COCO-DR Large outperforms the giant GPT-3 embedding model which has 500x more parameters. Our analysis show the correlation between COCO-DR's effectiveness in combating distribution shifts and improving zero-shot accuracy. Our code and model can be found at urlhttps://github.com/OpenMatch/COCO-DR.