# Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision

arXiv: https://arxiv.org/abs/2107.08582 · HTML: https://mlsanity.com/paper/2107.08582/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task](https://mlsanity.com/paper/1606.02858/) (2016)
- [Unsupervised Question Answering by Cloze Translation](https://mlsanity.com/paper/1906.04980/) (2019)
- [Machine Comprehension Using Match-LSTM and Answer Pointer](https://mlsanity.com/paper/1608.07905/) (2016)
- [What do Models Learn from Question Answering Datasets?](https://mlsanity.com/paper/2004.03490/) (2020)
- [Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering](https://mlsanity.com/paper/2004.11892/) (2020)
- [Unsupervised Multi-hop Question Answering by Question Generation](https://mlsanity.com/paper/2010.12623/) (2020)
- [Leveraging Knowledge Bases in LSTMs for Improving Machine Reading](https://mlsanity.com/paper/1902.09091/) (2019)
- [Span Selection Pre-training for Question Answering](https://mlsanity.com/paper/1909.04120/) (2019)
- [Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialogues](https://mlsanity.com/paper/2009.06265/) (2020)
- [Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering](https://mlsanity.com/paper/2005.00316/) (2020)
- [Simple and Effective Semi-Supervised Question Answering](https://mlsanity.com/paper/1804.00720/) (2018)
- [Harvesting and Refining Question-Answer Pairs for Unsupervised QA](https://mlsanity.com/paper/2005.02925/) (2020)
- [MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale](https://mlsanity.com/paper/2010.00980/) (2020)
- [Self-Supervised Dialogue Learning](https://mlsanity.com/paper/1907.00448/) (2019)
- [Improving Limited Labeled Dialogue State Tracking with Self-Supervision](https://mlsanity.com/paper/2010.13920/) (2020)
- [Unsupervised Domain Adaptation on Reading Comprehension](https://mlsanity.com/paper/1911.06137/) (2019)

## Built on by (most cited first, 3 of 3)

- [Learning to Retrieve Passages without Supervision](https://mlsanity.com/paper/2112.07708/) (2021)
- [ProQA: Structural Prompt-based Pre-training for Unified Question Answering](https://mlsanity.com/paper/2205.04040/) (2022)
- [Learning In-context Learning for Named Entity Recognition](https://mlsanity.com/paper/2305.11038/) (2023)

## Abstract (arXiv)

Despite recent success in machine reading comprehension (MRC), learning high-quality MRC models still requires large-scale labeled training data, even using strong pre-trained language models (PLMs). The pre-training tasks for PLMs are not question-answering or MRC-based tasks, making existing PLMs unable to be directly used for unsupervised MRC. Specifically, MRC aims to spot an accurate answer span from the given document, but PLMs focus on token filling in sentences. In this paper, we propose a new framework for unsupervised MRC. Firstly, we propose to learn to spot answer spans in documents via self-supervised learning, by designing a self-supervision pretext task for MRC - Spotting-MLM. Solving this task requires capturing deep interactions between sentences in documents. Secondly, we apply a simple sentence rewriting strategy in the inference stage to alleviate the expression mismatch between questions and documents. Experiments show that our method achieves a new state-of-the-art performance for unsupervised MRC.