mlsanityData to 7 Oct 2026

UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers

arXiv 2303.00807 ↗First posted 1 Mar 2023cs.IR cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Many information retrieval tasks require large labeled datasets for fine-tuning. However, such datasets are often unavailable, and their utility for real-world applications can diminish quickly due to domain shifts. To address this challenge, we develop and motivate a method for using large language models (LLMs) to generate large numbers of synthetic queries cheaply. The method begins by generating a small number of synthetic queries using an expensive LLM. After that, a much less expensive one is used to create large numbers of synthetic queries, which are used to fine-tune a family of reranker models. These rerankers are then distilled into a single efficient retriever for use in the target domain. We show that this technique boosts zero-shot accuracy in long-tail domains and achieves substantially lower latency than standard reranking methods.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 41 of its 57 references that are in the map. First posted on arXiv 1 Mar 2023.

Builds on

41 of 57 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
29 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  4. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  5. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  6. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  7. 2019Passage Re-ranking with BERTcited by 629
  8. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  9. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  10. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  11. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  12. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  13. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  14. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  15. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  16. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  17. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 143
  18. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 136
  19. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  20. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  21. 2022Task-aware Retrieval with Instructionscited by 107
  22. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  23. 2023InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrievalcited by 73
  24. 2021Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrievalcited by 63
  25. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
  26. 2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42
  27. 2021Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representationscited by 27
  28. 2022To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answeringcited by 5
  29. 2022Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarkingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 18; 2025: 4; 2026: 6.

37 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  5. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  6. 2023Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Costcited by 33
  7. 2023Large Language Model Augmented Narrative Driven Recommendationscited by 31
  8. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generationcited by 27
  9. 2023Large Language Models are Strong Zero-Shot Retrievercited by 25
  10. 2023Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrievalcited by 24
  11. 2024Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERTcited by 21
  12. 2024Leveraging Large Language Models for Relevance Judgments in Legal Case Retrievalcited by 16
25 more
  1. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  2. 2024Retrieval-Augmented Data Augmentation for Low-Resource Domain Taskscited by 12
  3. 2024Unsupervised Distractor Generation via Large Language Model Distilling and Counterfactual Contrastive Decodingcited by 11
  4. 2024KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Modelscited by 6
  5. 2025Improving Scientific Document Retrieval with Concept Coverage-based Query Set Generationcited by 6
  6. 2023Utilizing ChatGPT to Enhance Clinical Trial Enrollmentcited by 4
  7. 2024Leveraging Large Language Models for Entity Matchingcited by 3
  8. 2024Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labelscited by 3
  9. 2024Disentangling Questions from Query Generation for Task-Adaptive Retrievalcited by 3
  10. 2024Dynamic Uncertainty Ranking: Enhancing Retrieval-Augmented In-Context Learning for Long-Tail Knowledge in LLMscited by 3
  11. 2024Large Language Model as a Teacher for Zero-shot Tagging at Extreme Scalescited by 2
  12. 2024AutoPLC: Generating Vendor-Aware Structured Text for Programmable Logic Controllerscited by 2
  13. 2026IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answeringcited by 2
  14. 2023Prompt Generate Train (PGT): Few-shot Domain Adaption of Retrieval Augmented Generation Models for Open Book Question-Answeringcited by 1
  15. 2023Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoderscited by 1
  16. 2024DeliLaw: A Chinese Legal Counselling System Based on a Large Language Modelcited by 1
  17. 2024Are LLMs Effective Backbones for Fine-tuning? An Experimental Investigation of Supervised LLMs on Chinese Short Text Matchingcited by 0
  18. 2024LargePiG: Your Large Language Model is Secretly a Pointer Generatorcited by 0
  19. 2025ConceptCarve: Dynamic Realization of Evidencecited by 0
  20. 2025Evaluating the Robustness of Dense Retrievers in Interdisciplinary Domainscited by 0
  21. 2026Improving Scientific Document Retrieval with Academic Concept Indexcited by 0
  22. 2026RoIt-XMASA: Multi-Domain Multilingual Sentiment Analysis Dataset for Romanian and Italiancited by 0
  23. 2026Test-Time Training for Zero-Resource Dense Retrieval Rerankingcited by 0
  24. 2026PETRA: Transforming Web Text for Petroleum-Engineering Domain Adaptationcited by 0
  25. 2026Training Specialist Models without Reasoning Trajectories for Domain Expert Distillationcited by 0