mlsanityData to 7 Oct 2026

GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

arXiv 2112.07577 ↗First posted 14 Dec 2021cs.CL cs.IRcited by 119

Not on the map. Why?

Abstract (arXiv)

Dense retrieval approaches can overcome the lexical gap and lead to significantly improved search results. However, they require large amounts of training data which is not available for most domains. As shown in previous work (Thakur et al., 2021b), the performance of dense retrievers severely degrades under a domain shift. This limits the usage of dense retrieval approaches to only a few domains with large training datasets. In this paper, we propose the novel unsupervised domain adaptation method Generative Pseudo Labeling (GPL), which combines a query generator with pseudo labeling from a cross-encoder. On six representative domain-specialized datasets, we find the proposed GPL can outperform an out-of-the-box state-of-the-art dense retrieval approach by up to 9.3 points nDCG@10. GPL requires less (unlabeled) data from the target domain and is more robust in its training than previous methods. We further investigate the role of six recent pre-training methods in the scenario of domain adaptation for retrieval tasks, where only three could yield improved results. The best approach, TSDAE (Wang et al., 2021) can be combined with GPL, yielding another average improvement of 1.4 points nDCG@10 across the six tasks. The code and the models are available at https://github.com/UKPLab/gpl.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 14 Dec 2021.

Builds on

32 of 50 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  11. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
20 more
  1. 2019Passage Re-ranking with BERTcited by 629
  2. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  3. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  4. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 228
  5. 2019Document Expansion by Query Predictioncited by 225
  6. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
  7. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  8. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  9. 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 164
  10. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  11. 2020Embedding-based Retrieval in Facebook Searchcited by 155
  12. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 143
  13. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 136
  14. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  15. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  16. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  17. 2021UDALM: Unsupervised Domain Adaptation through Language Modelingcited by 33
  18. 2021Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representationscited by 27
  19. 2020Embedding-based Zero-shot Retrieval through Query Generationcited by 24
  20. 2020Semantic Product Search for Matching Structured Product Catalogs in E-Commercecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 27; 2023: 32; 2024: 18; 2025: 24; 2026: 18.

119 papers.

Sort:

By area:

  1. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  2. 2023Improving Text Embeddings with Large Language Modelscited by 295
  3. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  4. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  5. 2022Task-aware Retrieval with Instructionscited by 107
  6. 2023Dense X Retrieval: What Retrieval Granularity Should We Use?cited by 88
  7. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 63
  8. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
  9. 2022Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generationcited by 47
  10. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  11. 2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
  12. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
38 more
  1. 2023UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankerscited by 37
  2. 2023Rethinking the Role of Token Retrieval in Multi-Vector Retrievalcited by 28
  3. 2024Scaling Laws For Dense Retrievalcited by 28
  4. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
  5. 2023CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoningcited by 25
  6. 2023When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasetscited by 25
  7. 2023Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrievalcited by 24
  8. 2022A Thorough Examination on Zero-shot Dense Retrievalcited by 21
  9. 2022In Defense of Cross-Encoders for Zero-Shot Retrievalcited by 21
  10. 2022COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learningcited by 19
  11. 2022No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrievalcited by 17
  12. 2025Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMscited by 17
  13. 2022DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Enginecited by 15
  14. 2024Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspectivecited by 15
  15. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  16. 2023InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankerscited by 14
  17. 2023RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Modelscited by 13
  18. 2024RouterRetriever: Routing over a Mixture of Expert Embedding Modelscited by 12
  19. 2023Synergistic Interplay between Search and Large Language Models for Information Retrievalcited by 11
  20. 2023BERM: Training the Balanced and Extractable Representation for Matching to Improve Generalization Ability of Dense Retrievalcited by 11
  21. 2023Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answeringcited by 10
  22. 2022Query-as-context Pre-training for Dense Passage Retrievalcited by 9
  23. 2023Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answeringcited by 9
  24. 2024textttMixGR: Enhancing Retriever Generalization for Scientific Domain through Complementary Granularitycited by 9
  25. 2023Dense Retrieval Adaptation using Target Domain Descriptioncited by 8
  26. 2024Beyond Borders: Investigating Cross-Jurisdiction Transfer in Legal Case Summarizationcited by 8
  27. 2022A Study on the Efficiency and Generalization of Light Hybrid Retrieverscited by 7
  28. 2023Selecting which Dense Retriever to use for Zero-Shot Searchcited by 7
  29. 2024Reward-RAG: Enhancing RAG with Reward Driven Supervisioncited by 7
  30. 2022Knowledge Transfer from Answer Ranking to Answer Generationcited by 6
  31. 2023Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memoriescited by 6
  32. 2023Improving Content Retrievability in Search with Controllable Query Generationcited by 6
  33. 2023Compositional Zero-Shot Domain Transfer with Text-to-Text Modelscited by 6
  34. 2025Improving Scientific Document Retrieval with Concept Coverage-based Query Set Generationcited by 6
  35. 2022PseudoReasoner: Leveraging Pseudo Labels for Commonsense Knowledge Base Populationcited by 5
  36. 2022AugCSE: Contrastive Sentence Embedding with Diverse Augmentationscited by 5
  37. 2023QUADRo: Dataset and Models for QUestion-Answer Database Retrievalcited by 5
  38. 2023Generative User-Experience Research for Developing Domain-specific Natural Language Processing Applicationscited by 5

50 most cited of 119 All 119 (JSON)