# C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References

arXiv: https://arxiv.org/abs/2203.08928 · HTML: https://mlsanity.com/paper/2203.08928/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [A Discrete Hard EM Approach for Weakly Supervised Question Answering](https://mlsanity.com/paper/1909.04849/) (2019)
- [End-to-End Training of Neural Retrievers for Open-Domain Question Answering](https://mlsanity.com/paper/2101.00408/) (2021)
- [Unsupervised Question Answering by Cloze Translation](https://mlsanity.com/paper/1906.04980/) (2019)
- [Harvesting and Refining Question-Answer Pairs for Unsupervised QA](https://mlsanity.com/paper/2005.02925/) (2020)

## Built on by (most cited first, 4 of 4)

- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [Unifying Corroborative and Contributive Attributions in Large Language Models](https://mlsanity.com/paper/2311.12233/) (2023)
- [Domain Adaptation for Question Answering via Question Classification](https://mlsanity.com/paper/2209.04998/) (2022)
- [Domain Adaptation of Multilingual Semantic Search -- Literature Review](https://mlsanity.com/paper/2402.02932/) (2024)

## Abstract (arXiv)

We consider the problem of pretraining a two-stage open-domain question answering (QA) system (retriever + reader) with strong transfer capabilities. The key challenge is how to construct a large amount of high-quality question-answer-context triplets without task-specific annotations. Specifically, the triplets should align well with downstream tasks by: (i) covering a wide range of domains (for open-domain applications), (ii) linking a question to its semantically relevant context with supporting evidence (for training the retriever), and (iii) identifying the correct answer in the context (for training the reader). Previous pretraining approaches generally fall short of one or more of these requirements. In this work, we automatically construct a large-scale corpus that meets all three criteria by consulting millions of references cited within Wikipedia. The well-aligned pretraining signals benefit both the retriever and the reader significantly. Our pretrained retriever leads to 2%-10% absolute gains in top-20 accuracy. And with our pretrained reader, the entire system improves by up to 4% in exact match.