# Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need

arXiv: https://arxiv.org/abs/2108.09346 · HTML: https://mlsanity.com/paper/2108.09346/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2021.

## Built on by (most cited first, 15 of 15)

- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks](https://mlsanity.com/paper/2208.07652/) (2022)
- [A Survey on Deep Tabular Learning](https://mlsanity.com/paper/2410.12034/) (2024)
- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [Data-Efficient Autoregressive Document Retrieval for Fact Verification](https://mlsanity.com/paper/2211.09388/) (2022)
- [Leveraging Large Language Models for Effective Label-free Node Classification in Text-Attributed Graphs](https://mlsanity.com/paper/2412.11983/) (2024)
- [MCP: Self-supervised Pre-training for Personalized Chatbots with Multi-level Contrastive Sampling](https://mlsanity.com/paper/2210.08753/) (2022)
- [An Experimental Study on Pretraining Transformers from Scratch for IR](https://mlsanity.com/paper/2301.10444/) (2023)
- [Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval](https://mlsanity.com/paper/2302.01626/) (2023)
- [Pre-training for Information Retrieval: Are Hyperlinks Fully Explored?](https://mlsanity.com/paper/2209.06583/) (2022)
- [A thorough benchmark of automatic text classification: From traditional approaches to large language models](https://mlsanity.com/paper/2504.01930/) (2025)
- [Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition](https://mlsanity.com/paper/2505.07166/) (2025)
- [Unsupervised Dense Retrieval Training with Web Anchors](https://mlsanity.com/paper/2305.05834/) (2023)
- [On Elastic Language Models](https://mlsanity.com/paper/2311.07204/) (2023)
- [Coarse-Tuning for Ad-hoc Document Retrieval Using Pre-trained Language Models](https://mlsanity.com/paper/2403.16915/) (2024)

## Abstract (arXiv)

Designing pre-training objectives that more closely resemble the downstream tasks for pre-trained language models can lead to better performance at the fine-tuning stage, especially in the ad-hoc retrieval area. Existing pre-training approaches tailored for IR tried to incorporate weak supervised signals, such as query-likelihood based sampling, to construct pseudo query-document pairs from the raw textual corpus. However, these signals rely heavily on the sampling method. For example, the query likelihood model may lead to much noise in the constructed pre-training data. blfootnotedagger This work was done during an internship at Huawei. In this paper, we propose to leverage the large-scale hyperlinks and anchor texts to pre-train the language model for ad-hoc retrieval. Since the anchor texts are created by webmasters and can usually summarize the target document, it can help to build more accurate and reliable pre-training samples than a specific algorithm. Considering different views of the downstream ad-hoc retrieval, we devise four pre-training tasks based on the hyperlinks. We then pre-train the Transformer model to predict the pair-wise preference, jointly with the Masked Language Model objective. Experimental results on two large-scale ad-hoc retrieval datasets show the significant improvement of our model compared with the existing methods.