From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Model to Pre-trained Machine Reader
Not on the map. Why?
Abstract (arXiv)
We present Pre-trained Machine Reader (PMR), a novel method for retrofitting pre-trained masked language models (MLMs) to pre-trained machine reading comprehension (MRC) models without acquiring labeled data. PMR can resolve the discrepancy between model pre-training and downstream fine-tuning of existing MLMs. To build the proposed PMR, we constructed a large volume of general-purpose and high-quality MRC-style training data by using Wikipedia hyperlinks and designed a Wiki Anchor Extraction task to guide the MRC-style pre-training. Apart from its simplicity, PMR effectively solves extraction tasks, such as Extractive Question Answering and Named Entity Recognition. PMR shows tremendous improvements over existing approaches, especially in low-resource scenarios. When applied to the sequence classification task in the MRC formulation, PMR enables the extraction of high-quality rationales to explain the classification process, thereby providing greater prediction explainability. PMR also has the potential to serve as a unified model for tackling various extraction and classification tasks in the MRC formulation.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 49 of its 71 references that are in the map. First posted on arXiv 9 Dec 2022.
Builds on
49 of 71 references.
References · 49
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
37 more
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
- 2015Teaching Machines to Read and Comprehendcited by 1,039
- 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
- 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
- 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
- 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
- 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
- 2021GPT Understands, Toocited by 715
- 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
- 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
- 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
- 2016NewsQA: A Machine Comprehension Datasetcited by 330
- 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
- 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
- 2022Unified Structure Generation for Universal Information Extractioncited by 220
- 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
- 2015Named Entity Recognition with Bidirectional LSTM-CNNscited by 201
- 2019A Unified MRC Framework for Named Entity Recognitioncited by 199
- 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
- 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
- 2019DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehensioncited by 156
- 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
- 2021Template-Based Named Entity Recognition Using BARTcited by 153
- 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
- 2021A Unified Generative Framework for Various NER Subtaskscited by 141
- 2019Coreference Resolution as Query-based Span Predictioncited by 91
- 2021Improving Named Entity Recognition by External Context Retrieving and Cooperative Learningcited by 76
- 2021Template-free Prompt Tuning for Few-shot NERcited by 73
- 2018DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehensioncited by 70
- 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
- 2019Unsupervised Question Answering by Cloze Translationcited by 59
- 2019Span Selection Pre-training for Question Answeringcited by 36
- 2021LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Promptingcited by 27
- 2021FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text modelscited by 26
- 2022ProQA: Structural Prompt-based Pre-training for Unified Question Answeringcited by 21
- 2022PeerDA: Data Augmentation via Modeling Peer Relation for Span Identification Taskscited by 7
- 2021Weakly Supervised Pre-Training for Multi-Hop Retrievercited by 3
Built on by
Citing papers per year: 2023: 2.
2 papers.