# MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension

arXiv: https://arxiv.org/abs/1910.09753 · HTML: https://mlsanity.com/paper/1910.09753/

## Key facts

- Cited by 184 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2019.

## Built on by (most cited first, 30 of 184)

- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks](https://mlsanity.com/paper/2206.08916/) (2022)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://mlsanity.com/paper/2305.13169/) (2023)
- [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://mlsanity.com/paper/2201.03533/) (2022)
- [Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation](https://mlsanity.com/paper/2307.11019/) (2023)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence](https://mlsanity.com/paper/2210.13701/) (2022)
- [Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2303.02861/) (2023)
- [CommonsenseQA 2.0: Exposing the Limits of AI through Gamification](https://mlsanity.com/paper/2201.05320/) (2022)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"](https://mlsanity.com/paper/2410.03727/) (2024)
- [FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks](https://mlsanity.com/paper/2104.08815/) (2021)
- [ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts](https://mlsanity.com/paper/2205.11961/) (2022)
- [Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation](https://mlsanity.com/paper/2104.08678/) (2021)
- [Few-Shot Question Answering by Pretraining Span Selection](https://mlsanity.com/paper/2101.00438/) (2021)
- [Comparing Test Sets with Item Response Theory](https://mlsanity.com/paper/2106.00840/) (2021)
- [Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking](https://mlsanity.com/paper/2106.06052/) (2021)
- [Zero-Shot Dialogue State Tracking via Cross-Task Transfer](https://mlsanity.com/paper/2109.04655/) (2021)
- [Resolving Knowledge Conflicts in Large Language Models](https://mlsanity.com/paper/2310.00935/) (2023)
- [Universal Information Extraction as Unified Semantic Matching](https://mlsanity.com/paper/2301.03282/) (2023)
- [Knowing More About Questions Can Help: Improving Calibration in Question Answering](https://mlsanity.com/paper/2106.01494/) (2021)
- [Read before Generate! Faithful Long Form Question Answering with Machine Reading](https://mlsanity.com/paper/2203.00343/) (2022)
- [500xCompressor: Generalized Prompt Compression for Large Language Models](https://mlsanity.com/paper/2408.03094/) (2024)
- [Can NLI Models Verify QA Systems' Predictions?](https://mlsanity.com/paper/2104.08731/) (2021)

## Abstract (arXiv)

We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems. In this task, we adapted and unified 18 distinct question answering datasets into the same format. Among them, six datasets were made available for training, six datasets were made available for development, and the final six were hidden for final evaluation. Ten teams submitted systems, which explored various ideas including data sampling, multi-task learning, adversarial training and ensembling. The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than our initial baseline based on BERT.