mlsanityData to 7 Oct 2026

Towards Robust Neural Retrieval Models with Synthetic Pre-Training

arXiv 2104.07800 ↗First posted 15 Apr 2021cs.CL cs.AI cs.IRcited by 9

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that commonly available machine reading comprehension (MRC) datasets can be used to train high-performance neural information retrieval (IR) systems. However, the evaluation of neural IR has so far been limited to standard supervised learning settings, where they have outperformed traditional term matching baselines. We conduct in-domain and out-of-domain evaluations of neural IR, and seek to improve its robustness across different scenarios, including zero-shot settings. We show that synthetic training examples generated using a sequence-to-sequence generator can be effective towards this goal: in our experiments, pre-training with synthetic examples improves retrieval performance in both in-domain and out-of-domain evaluation on five different test sets.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 16 of its 25 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

16 of 25 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  10. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  11. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  12. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
4 more
  1. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  2. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  3. 2020End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systemscited by 65
  4. 2020Multi-Stage Pre-training for Low-Resource Domain Adaptationcited by 24

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 4; 2023: 2; 2024: 2.

9 papers.

  1. 2021Learning to Retrieve Passages without Supervisioncited by 46
  2. 2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28
  3. 2023Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answeringcited by 15
  4. 2024Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspectivecited by 15
  5. 2022Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answeringcited by 14
  6. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  7. 2022Entity-Conditioned Question Generation for Robust Attention Distribution in Neural Information Retrievalcited by 5
  8. 2023Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrievalcited by 3
  9. 2024Domain Adaptation of Multilingual Semantic Search -- Literature Reviewcited by 0