An Experimental Study on Pretraining Transformers from Scratch for IR
Not on the map. Why?
Abstract (arXiv)
Finetuning Pretrained Language Models (PLM) for IR has been de facto the standard practice since their breakthrough effectiveness few years ago. But, is this approach well understood? In this paper, we study the impact of the pretraining collection on the final IR effectiveness. In particular, we challenge the current hypothesis that PLM shall be trained on a large enough generic collection and we show that pretraining from scratch on the collection of interest is surprisingly competitive with the current approach. We benchmark first-stage ranking rankers and cross-encoders for reranking on the task of general passage retrieval on MSMARCO, Mr-Tydi for Arabic, Japanese and Russian, and TripClick for specific domain. Contrary to popular belief, we show that, for finetuning first-stage rankers, models pretrained solely on their collection have equivalent or better effectiveness compared to more general models. However, there is a slight effectiveness drop for rerankers pretrained only on the target collection. Overall, our study sheds a new light on the role of the pretraining collection and should make our community ponder on building specialized models by pretraining from scratch. Last but not least, doing so could enable better control of efficiency, data bias and replicability, which are key research questions for the IR community.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 25 Jan 2023.
Builds on
45 of 62 references.
References · 45
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
- 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
- 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
33 more
- 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
- 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
- 2019Passage Re-ranking with BERTcited by 645
- 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
- 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
- 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 267
- 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
- 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
- 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
- 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
- 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
- 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
- 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 145
- 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 119
- 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
- 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
- 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
- 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
- 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
- 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
- 2021A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniquescited by 69
- 2017Neural Ranking Models with Weak Supervisioncited by 61
- 2020Minimizing FLOPs to Learn Efficient Sparse Representationscited by 46
- 2020PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 40
- 2022Transfer Learning Approaches for Building Cross-Language Dense Retrieval Modelscited by 36
- 2021Are Pre-trained Convolutions Better than Pre-trained Transformers?cited by 27
- 2022An Efficiency Study for SPLADE Modelscited by 26
- 2021B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 25
- 2019On the use of BERT for Neural Machine Translationcited by 23
- 2020On Losses for Modern Language Modelscited by 21
- 2021TripClick: The Log Files of a Large Health Web Search Enginecited by 18
- 2021Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Needcited by 15
- 2022Establishing Strong Baselines for TripClick Health Retrievalcited by 4
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 1; 2026: 1.
3 papers.