mlsanityData to 7 Oct 2026

B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval

arXiv 2104.09791 ↗First posted 20 Apr 2021cs.IRcited by 25

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Pre-training and fine-tuning have achieved remarkable success in many downstream natural language processing (NLP) tasks. Recently, pre-training methods tailored for information retrieval (IR) have also been explored, and the latest success is the PROP method which has reached new SOTA on a variety of ad-hoc retrieval benchmarks. The basic idea of PROP is to construct the textitrepresentative words prediction (ROP) task for pre-training inspired by the query likelihood model. Despite its exciting performance, the effectiveness of PROP might be bounded by the classical unigram language model adopted in the ROP task construction process. To tackle this problem, we propose a bootstrapped pre-training method (namely B-PROP) based on BERT for ad-hoc retrieval. The key idea is to use the powerful contextual language model BERT to replace the classical unigram language model for the ROP task construction, and re-train BERT itself towards the tailored objective for IR. Specifically, we introduce a novel contrastive method, inspired by the divergence-from-randomness idea, to leverage BERT's self-attention mechanism to sample representative words from the document. By further fine-tuning on downstream ad-hoc retrieval tasks, our method achieves significant improvements over baselines without pre-training or with other pre-training methods, and further pushes forward the SOTA on a variety of ad-hoc retrieval tasks.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 30 of its 47 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

30 of 47 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  11. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  12. 2019Passage Re-ranking with BERTcited by 629
18 more
  1. 2016Neural Architectures for Named Entity Recognitioncited by 548
  2. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  3. 2020Overview of the TREC 2019 deep learning trackcited by 282
  4. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  5. 2019Document Expansion by Query Predictioncited by 225
  6. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  7. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  8. 2017A Deep Relevance Matching Model for Ad-hoc Retrievalcited by 126
  9. 2013Introducing LETOR 4.0 Datasetscited by 120
  10. 2019Deeper Text Understanding for IR with Contextual Neural Language Modelingcited by 111
  11. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  12. 2020PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 40
  13. 2019Span Selection Pre-training for Question Answeringcited by 36
  14. 2019Simple Applications of BERT for Ad Hoc Document Retrievalcited by 34
  15. 2017DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrievalcited by 21
  16. 2019MatchZoo: A Learning, Practicing, and Developing System for Neural Text Matchingcited by 17
  17. 2018Modeling Diverse Relevance Patterns in Ad-hoc Retrievalcited by 10
  18. 2021A Linguistic Study on Relevance Modeling in Information Retrievalcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 8; 2023: 10; 2024: 2; 2025: 4.

25 papers.

Sort:

By area:

  1. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 77
  2. 2022CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Taskscited by 36
  3. 2022Fine-Grained Distillation for Long Document Retrievalcited by 29
  4. 2022Towards Best Practices for Training Multilingual Dense Retrieval Modelscited by 26
  5. 2022HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Rerankingcited by 19
  6. 2024Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decodingcited by 19
  7. 2023Agent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting Using Multi-agent LLMcited by 18
  8. 2023Wikiformer: Pre-training with Structured Information of Wikipedia for Ad-hoc Retrievalcited by 15
  9. 2023RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Modelscited by 13
  10. 2023QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Searchcited by 10
  11. 2022P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuningcited by 7
  12. 2023Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Offcited by 7
13 more
  1. 2025Hypencoder: Hypernetworks for Information Retrievalcited by 7
  2. 2025Unleashing the Power of LLMs in Dense Retrieval with Query Likelihood Modelingcited by 5
  3. 2025Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generationcited by 5
  4. 2022Retrieval Oriented Masking Pre-training Language Model for Dense Passage Retrievalcited by 4
  5. 2022AugTriever: Unsupervised Dense Retrieval and Domain Adaptation by Scalable Data Augmentationcited by 3
  6. 2023An Experimental Study on Pretraining Transformers from Scratch for IRcited by 3
  7. 2023Challenging Decoder helps in Masked Auto-Encoder Pre-training for Dense Passage Retrievalcited by 3
  8. 2022Pre-training for Information Retrieval: Are Hyperlinks Fully Explored?cited by 2
  9. 2021YES SIR!Optimizing Semantic Space of Negatives with Self-Involvement Rankercited by 1
  10. 2023Generate to Understand for Representationcited by 0
  11. 2023On Elastic Language Modelscited by 0
  12. 2024Coarse-Tuning for Ad-hoc Document Retrieval Using Pre-trained Language Modelscited by 0
  13. 2025Text2Token: Unsupervised Text Representation Learning with Token Target Predictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
  2. 2023Zero-shot Retrieval: Augmenting Pre-trained Models with Search Engines
  3. 2024Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval
  4. 2026DiffRetriever: Parallel Representative Tokens for Retrieval with Diffusion Language Models
  5. 2023CoT-BERT: Enhancing Unsupervised Sentence Representation through Chain-of-Thought
  6. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context
  7. 2025ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Reranking
  8. 2026DEO: Training-Free Direct Embedding Optimization for Negation-Aware Retrieval