# Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering

arXiv: https://arxiv.org/abs/2203.06942 · HTML: https://mlsanity.com/paper/2203.06942/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://mlsanity.com/paper/2102.07033/) (2021)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [Pre-training Tasks for Embedding-based Large-scale Retrieval](https://mlsanity.com/paper/2002.03932/) (2020)
- [End-to-End Training of Neural Retrievers for Open-Domain Question Answering](https://mlsanity.com/paper/2101.00408/) (2021)
- [Neural Ranking Models with Weak Supervision](https://mlsanity.com/paper/1704.08803/) (2017)
- [Domain-matched Pre-training Tasks for Dense Retrieval](https://mlsanity.com/paper/2107.13602/) (2021)
- [Towards Robust Neural Retrieval Models with Synthetic Pre-Training](https://mlsanity.com/paper/2104.07800/) (2021)
- [Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap](https://mlsanity.com/paper/2109.11256/) (2021)

## Built on by (most cited first, 15 of 15)

- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Questions Are All You Need to Train a Dense Passage Retriever](https://mlsanity.com/paper/2206.10658/) (2022)
- [Large Language Models are Strong Zero-Shot Retriever](https://mlsanity.com/paper/2304.14233/) (2023)
- [UnifieR: A Unified Retriever for Large-Scale Retrieval](https://mlsanity.com/paper/2205.11194/) (2022)
- [Chain-of-Skills: A Configurable Model for Open-domain Question Answering](https://mlsanity.com/paper/2305.03130/) (2023)
- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [Synergistic Interplay between Search and Large Language Models for Information Retrieval](https://mlsanity.com/paper/2305.07402/) (2023)
- [Weaker Than You Think: A Critical Look at Weakly Supervised Learning](https://mlsanity.com/paper/2305.17442/) (2023)
- [MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrievers](https://mlsanity.com/paper/2212.07841/) (2022)
- [OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learning](https://mlsanity.com/paper/2503.08398/) (2025)
- [Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval](https://mlsanity.com/paper/2302.01626/) (2023)
- [Optimizing Retrieval for RAG via Reinforcement Learning](https://mlsanity.com/paper/2510.24652/) (2025)
- [Semi-Parametric Retrieval via Binary Bag-of-Tokens Index](https://mlsanity.com/paper/2405.01924/) (2024)
- [WRAP++: Web discoveRy Amplified Pretraining](https://mlsanity.com/paper/2604.06829/) (2026)
- [AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining](https://mlsanity.com/paper/2610.20086/) (2026)

## Abstract (arXiv)

To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR). However, there still remains a large discrepancy between the provided upstream signals and the downstream question-passage relevance, which leads to less improvement. To bridge this gap, we propose the HyperLink-induced Pre-training (HLP), a method to pre-train the dense retriever with the text relevance induced by hyperlink-based topology within Web documents. We demonstrate that the hyperlink-based structures of dual-link and co-mention can provide effective relevance signals for large-scale pre-training that better facilitate downstream passage retrieval. We investigate the effectiveness of our approach across a wide range of open-domain QA datasets under zero-shot, few-shot, multi-hop, and out-of-domain scenarios. The experiments show our HLP outperforms the BM25 by up to 7 points as well as other pre-training methods by more than 10 points in terms of top-20 retrieval accuracy under the zero-shot scenario. Furthermore, HLP significantly outperforms other pre-training methods under the other scenarios.