mlsanityData to 7 Oct 2026

COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning

arXiv 2210.15212 ↗First posted 27 Oct 2022cs.CL cs.IR cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

We present a new zero-shot dense retrieval (ZeroDR) method, COCO-DR, to improve the generalization ability of dense retrieval by combating the distribution shifts between source training tasks and target scenarios. To mitigate the impact of document differences, COCO-DR continues pretraining the language model on the target corpora to adapt the model to target distributions via COtinuous COtrastive learning. To prepare for unseen target queries, COCO-DR leverages implicit Distributionally Robust Optimization (iDRO) to reweight samples from different source query clusters for improving model robustness over rare queries during fine-tuning. COCO-DR achieves superior average performance on BEIR, the zero-shot retrieval benchmark. At BERT Base scale, COCO-DR Base outperforms other ZeroDR models with 60x larger size. At BERT Large scale, COCO-DR Large outperforms the giant GPT-3 embedding model which has 500x more parameters. Our analysis show the correlation between COCO-DR's effectiveness in combating distribution shifts and improving zero-shot accuracy. Our code and model can be found at urlhttps://github.com/OpenMatch/COCO-DR.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 48 of its 76 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

48 of 76 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
36 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  3. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  4. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  6. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  7. 2020In Search of Lost Domain Generalizationcited by 1,042
  8. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  9. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  10. 2019Green AIcited by 620
  11. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  12. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  13. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  14. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  15. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  16. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  17. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  18. 2020No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problemscited by 208
  19. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  20. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  21. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  22. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  23. 2019Distributionally Robust Language Modelingcited by 142
  24. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  25. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  26. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  27. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  28. 2021Examining and Combating Spurious Features under Distribution Shiftcited by 64
  29. 2021UDALM: Unsupervised Domain Adaptation through Language Modelingcited by 33
  30. 2021Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representationscited by 27
  31. 2021Focus on the Common Good: Group Distributional Robustness Followscited by 25
  32. 2022LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrievalcited by 24
  33. 2022Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Predictioncited by 23
  34. 2022A Thorough Examination on Zero-shot Dense Retrievalcited by 21
  35. 2021OpenMatch: An Open Source Library for Neu-IR Researchcited by 6
  36. 2021Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literaturecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 5; 2025: 3; 2026: 4.

19 papers.

Sort:

By area:

  1. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  2. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  3. 2025Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaborationcited by 19
  4. 2025Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidencecited by 16
  5. 2023Synergistic Interplay between Search and Large Language Models for Information Retrievalcited by 11
  6. 2023Chain-of-Factors Paper-Reviewer Matchingcited by 11
  7. 2023Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Trainingcited by 8
  8. 2024Beyond Borders: Investigating Cross-Jurisdiction Transfer in Legal Case Summarizationcited by 8
  9. 2023I3: Intent-Introspective Retrieval Conditioned on Instructionscited by 5
  10. 2023Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrievalcited by 4
  11. 2024Large Language Model Augmented Exercise Retrieval for Personalized Language Learningcited by 4
  12. 2024Investigating Mixture of Experts in Dense Retrievalcited by 4
7 more
  1. 2024PairDistill: Pairwise Relevance Distillation for Dense Retrievalcited by 3
  2. 2024DREditor: An Time-efficient Approach for Building a Domain-specific Dense Retrieval Modelcited by 2
  3. 2025A Representation Sharpening Framework for Zero Shot Dense Retrievalcited by 1
  4. 2026With Argus Eyes: Assessing Retrieval Gaps via Uncertainty Scoring to Detect and Remedy Retrieval Blind Spotscited by 1
  5. 2026Beyond Paper-to-Paper: Structured Profiling and Rubric Scoring for Paper-Reviewer Matchingcited by 0
  6. 2026CoHyDE: Iterative Co-Training of LLM Rewriter & Dense Encoder for Tool Retrievalcited by 0
  7. 2026When Does Dense Retrieval Need Asymmetric Geometry? A Bias-Variance Theory of Shared and Dual Projectionscited by 0