# Benchmarking Robustness of Machine Reading Comprehension Models

arXiv: https://arxiv.org/abs/2004.14004 · HTML: https://mlsanity.com/paper/2004.14004/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 18 of 18)

- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models](https://mlsanity.com/paper/2105.11136/) (2021)
- [Achieving Model Robustness through Discrete Adversarial Training](https://mlsanity.com/paper/2104.05062/) (2021)
- [On the Robustness of Reading Comprehension Models to Entity Renaming](https://mlsanity.com/paper/2110.08555/) (2021)
- [Contrastive Fine-tuning Improves Robustness for Neural Rankers](https://mlsanity.com/paper/2105.12932/) (2021)
- [Model-tuning Via Prompts Makes NLP Models Adversarially Robust](https://mlsanity.com/paper/2303.07320/) (2023)
- [Desiderata for the Context Use of Question Answering Systems](https://mlsanity.com/paper/2401.18001/) (2024)
- [How Fragile is Relation Extraction under Entity Replacements?](https://mlsanity.com/paper/2305.13551/) (2023)
- [A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension](https://mlsanity.com/paper/2209.01824/) (2022)
- [The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models](https://mlsanity.com/paper/2302.00094/) (2023)
- [PALRACE: Reading Comprehension Dataset with Human Data and Labeled Rationales](https://mlsanity.com/paper/2106.12373/) (2021)
- [Pay Attention to Real World Perturbations! Natural Robustness Evaluation in Machine Reading Comprehension](https://mlsanity.com/paper/2502.16523/) (2025)
- [READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises](https://mlsanity.com/paper/2302.07324/) (2023)
- [Using calibrator to improve robustness in Machine Reading Comprehension](https://mlsanity.com/paper/2202.11865/) (2022)
- [An Understanding-Oriented Robust Machine Reading Comprehension Model](https://mlsanity.com/paper/2207.00187/) (2022)
- [Evaluating the Robustness of Machine Reading Comprehension Models to Low Resource Entity Renaming](https://mlsanity.com/paper/2304.03145/) (2023)
- [Cross-corpus Readability Compatibility Assessment for English Texts](https://mlsanity.com/paper/2306.09704/) (2023)
- [AGent: A Novel Pipeline for Automatically Creating Unanswerable Questions](https://mlsanity.com/paper/2309.05103/) (2023)

## Abstract (arXiv)

Machine Reading Comprehension (MRC) is an important testbed for evaluating models' natural language understanding (NLU) ability. There has been rapid progress in this area, with new models achieving impressive performance on various benchmarks. However, existing benchmarks only evaluate models on in-domain test sets without considering their robustness under test-time perturbations or adversarial attacks. To fill this important gap, we construct AdvRACE (Adversarial RACE), a new model-agnostic benchmark for evaluating the robustness of MRC models under four different types of adversarial attacks, including our novel distractor extraction and generation attacks. We show that state-of-the-art (SOTA) models are vulnerable to all of these attacks. We conclude that there is substantial room for building more robust MRC models and our benchmark can help motivate and measure progress in this area. We release our data and code at https://github.com/NoviScl/AdvRACE .