mlsanityData to 7 Oct 2026

Relation-Guided Pre-Training for Open-Domain Question Answering

arXiv 2109.10346 ↗First posted 21 Sep 2021cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Answering complex open-domain questions requires understanding the latent relations between involving entities. However, we found that the existing QA datasets are extremely imbalanced in some types of relations, which hurts the generalization performance over questions with long-tail relations. To remedy this problem, in this paper, we propose a Relation-Guided Pre-Training (RGPT-QA) framework. We first generate a relational QA dataset covering a wide range of relations from both the Wikidata triplets and Wikipedia hyperlinks. We then pre-train a QA model to infer the latent relations from the question, and then conduct extractive QA to get the target answer entity. We demonstrate that by pretraining with propoed RGPT-QA techique, the popular open-domain QA model, Dense Passage Retriever (DPR), achieves 2.2%, 2.4%, and 6.3% absolute improvement in Exact Match accuracy on Natural Questions, TriviaQA, and WebQuestions. Particularly, we show that RGPT-QA improves significantly on questions with long-tail relations

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 24 of its 27 references that are in the map. First posted on arXiv 21 Sep 2021.

Builds on

24 of 27 references.

References · 24

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  5. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  6. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  7. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  8. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  9. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  10. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  11. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  12. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
12 more
  1. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  2. 2020Training Question Answering Models From Synthetic Datacited by 95
  3. 2019A Discrete Hard EM Approach for Weakly Supervised Question Answeringcited by 78
  4. 2019Unsupervised Question Answering by Cloze Translationcited by 59
  5. 2019Knowledge Guided Text Retrieval and Reading for Open Domain Question Answeringcited by 59
  6. 2020Differentiable Reasoning over a Virtual Knowledge Basecited by 39
  7. 2020Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answeringcited by 39
  8. 2020Unsupervised Multi-hop Question Answering by Question Generationcited by 38
  9. 2020Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledgecited by 37
  10. 2018Simple and Effective Semi-Supervised Question Answeringcited by 22
  11. 2020Harvesting and Refining Question-Answer Pairs for Unsupervised QAcited by 20
  12. 2021Reasoning Over Virtual Knowledge Bases With Open Predicate Relationscited by 18

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 4; 2023: 1; 2024: 1.

7 papers.

  1. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  2. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  3. 2022ProQA: Structural Prompt-based Pre-training for Unified Question Answeringcited by 21
  4. 2021You Only Need One Model for Open-domain Question Answeringcited by 17
  5. 2022Empowering Language Models with Knowledge Graph Reasoning for Question Answeringcited by 14
  6. 2023Pre-training Language Models for Comparative Reasoningcited by 6
  7. 2022Momentum Contrastive Pre-training for Question Answeringcited by 2