mlsanityData to 14 Oct 2026

Domain-matched Pre-training Tasks for Dense Retrieval

arXiv 2107.13602 ↗First posted 28 Jul 2021cs.CL cs.IRcited by 53

Not on the map. Why?

Abstract (arXiv)

Pre-training on larger datasets with ever increasing model size is now a proven recipe for increased performance across almost all NLP tasks. A notable exception is information retrieval, where additional pre-training has so far failed to produce convincing results. We show that, with the right pre-training setup, this barrier can be overcome. We demonstrate this by pre-training large bi-encoder models on 1) a recently released set of 65 million synthetically generated questions, and 2) 200 million post-comment pairs from a preexisting dataset of Reddit conversations made available by pushshift.io. We evaluate on a set of information retrieval and dialogue retrieval benchmarks, showing substantial improvements over supervised baselines.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 28 Jul 2021.

Builds on

38 of 50 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2017Billion-scale similarity search with GPUscited by 2,497
  8. 2018Deep contextualized word representationscited by 2,290
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  12. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
26 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  2. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  3. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  4. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  5. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  6. 2020Recipes for building an open-domain chatbotcited by 593
  7. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  8. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  9. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  10. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  11. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  12. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 200
  13. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  14. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  15. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 179
  16. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
  17. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  18. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  19. 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answeringcited by 75
  20. 2021Multi-task Retrieval for Knowledge-Intensive Taskscited by 44
  21. 2019Cloze-driven Pretraining of Self-attention Networkscited by 41
  22. 2020Is Retriever Merely an Approximator of Reader?cited by 30
  23. 2021Question Answering Infused Pre-training of General-Purpose Contextualized Representationscited by 19
  24. 2019Sequential Attention-based Network for Noetic End-to-End Response Selectioncited by 6
  25. 2018A Large-Scale Corpus for Conversation Disentanglementcited by 3
  26. 2018Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpuscited by 2

Built on by

Cited by year

Citing papers per year: 2021: 13; 2022: 24; 2023: 12; 2024: 1; 2025: 3.

53 papers.

Sort:

By area:

  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  3. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  4. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 235
  5. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  6. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  7. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 118
  8. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 113
  9. 2023Dense X Retrieval: What Retrieval Granularity Should We Use?cited by 88
  10. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 83
  11. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 60
  12. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
38 more
  1. 2021Learning to Retrieve Passages without Supervisioncited by 46
  2. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  3. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  4. 2022Towards Robust Ranker for Text Retrievalcited by 37
  5. 2021Open Domain Question Answering with A Unified Knowledge Interfacecited by 32
  6. 2023Rethinking the Role of Token Retrieval in Multi-Vector Retrievalcited by 31
  7. 2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28
  8. 2021Simple Local Attentions Remain Competitive for Long-Context Taskscited by 25
  9. 2022LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrievalcited by 24
  10. 2022PROD: Progressive Distillation for Dense Retrievalcited by 21
  11. 2022SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrievalcited by 21
  12. 2022Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answeringcited by 19
  13. 2023Chain-of-Skills: A Configurable Model for Open-domain Question Answeringcited by 19
  14. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  15. 2021MFAQ: a Multilingual FAQ Datasetcited by 17
  16. 2021You Only Need One Model for Open-domain Question Answeringcited by 17
  17. 2022Mixed-modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQAcited by 17
  18. 2022Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answeringcited by 15
  19. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  20. 2021CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Trainingcited by 13
  21. 2022Multi-Vector Retrieval as Sparse Alignmentcited by 12
  22. 2022Improving Wikipedia Verifiability with AIcited by 11
  23. 2023EmbedDistill: A Geometric Knowledge Distillation for Information Retrievalcited by 11
  24. 2021Boosted Dense Retrievercited by 10
  25. 2022DPTDR: Deep Prompt Tuning for Dense Passage Retrievalcited by 10
  26. 2024textttMixGR: Enhancing Retriever Generalization for Scientific Domain through Complementary Granularitycited by 9
  27. 2022Retrieve-and-Fill for Scenario-based Task-Oriented Semantic Parsingcited by 7
  28. 2022Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Modelscited by 6
  29. 2022Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negativescited by 5
  30. 2023Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiencycited by 5
  31. 2025RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasetscited by 5
  32. 2023Mitigating the Impact of False Negatives in Dense Retrieval with Contrastive Confidence Regularizationcited by 4
  33. 2022Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answeringcited by 3
  34. 2023Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrievalcited by 3
  35. 2023Noisy Self-Training with Synthetic Queries for Dense Retrievalcited by 3
  36. 2025Revela: Dense Retriever Learning via Language Modelingcited by 3
  37. 2022Towards Generalizable Semantic Product Search by Text Similarity Pre-training on Search Click Logscited by 2
  38. 2023Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Gamescited by 2

50 most cited of 53 All 53 (JSON)