# Relation-Guided Pre-Training for Open-Domain Question Answering

arXiv: https://arxiv.org/abs/2109.10346 · HTML: https://mlsanity.com/paper/2109.10346/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 27 references that are in the map.
- First posted on arXiv 21 Sep 2021.

## Builds on (most cited first, 24 of 24)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://mlsanity.com/paper/1911.10470/) (2019)
- [Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)](https://mlsanity.com/paper/1405.5869/) (2014)
- [Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model](https://mlsanity.com/paper/1912.09637/) (2019)
- [Training Question Answering Models From Synthetic Data](https://mlsanity.com/paper/2002.09599/) (2020)
- [A Discrete Hard EM Approach for Weakly Supervised Question Answering](https://mlsanity.com/paper/1909.04849/) (2019)
- [Unsupervised Question Answering by Cloze Translation](https://mlsanity.com/paper/1906.04980/) (2019)
- [Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering](https://mlsanity.com/paper/1911.03868/) (2019)
- [Differentiable Reasoning over a Virtual Knowledge Base](https://mlsanity.com/paper/2002.10640/) (2020)
- [Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering](https://mlsanity.com/paper/2004.11892/) (2020)
- [Unsupervised Multi-hop Question Answering by Question Generation](https://mlsanity.com/paper/2010.12623/) (2020)
- [Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge](https://mlsanity.com/paper/2007.00849/) (2020)
- [Simple and Effective Semi-Supervised Question Answering](https://mlsanity.com/paper/1804.00720/) (2018)
- [Harvesting and Refining Question-Answer Pairs for Unsupervised QA](https://mlsanity.com/paper/2005.02925/) (2020)
- [Reasoning Over Virtual Knowledge Bases With Open Predicate Relations](https://mlsanity.com/paper/2102.07043/) (2021)

## Built on by (most cited first, 7 of 7)

- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [ProQA: Structural Prompt-based Pre-training for Unified Question Answering](https://mlsanity.com/paper/2205.04040/) (2022)
- [You Only Need One Model for Open-domain Question Answering](https://mlsanity.com/paper/2112.07381/) (2021)
- [Empowering Language Models with Knowledge Graph Reasoning for Question Answering](https://mlsanity.com/paper/2211.08380/) (2022)
- [Pre-training Language Models for Comparative Reasoning](https://mlsanity.com/paper/2305.14457/) (2023)
- [Momentum Contrastive Pre-training for Question Answering](https://mlsanity.com/paper/2212.05762/) (2022)

## Abstract (arXiv)

Answering complex open-domain questions requires understanding the latent relations between involving entities. However, we found that the existing QA datasets are extremely imbalanced in some types of relations, which hurts the generalization performance over questions with long-tail relations. To remedy this problem, in this paper, we propose a Relation-Guided Pre-Training (RGPT-QA) framework. We first generate a relational QA dataset covering a wide range of relations from both the Wikidata triplets and Wikipedia hyperlinks. We then pre-train a QA model to infer the latent relations from the question, and then conduct extractive QA to get the target answer entity. We demonstrate that by pretraining with propoed RGPT-QA techique, the popular open-domain QA model, Dense Passage Retriever (DPR), achieves 2.2%, 2.4%, and 6.3% absolute improvement in Exact Match accuracy on Natural Questions, TriviaQA, and WebQuestions. Particularly, we show that RGPT-QA improves significantly on questions with long-tail relations