# The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models

arXiv: https://arxiv.org/abs/2302.00094 · HTML: https://mlsanity.com/paper/2302.00094/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 59 references that are in the map.
- First posted on arXiv 31 Jan 2023.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [Generating Natural Adversarial Examples](https://mlsanity.com/paper/1710.11342/) (2017)
- [Event Extraction by Answering (Almost) Natural Questions](https://mlsanity.com/paper/2004.13625/) (2020)
- [End-to-End Open-Domain Question Answering with BERTserini](https://mlsanity.com/paper/1902.01718/) (2019)
- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [Deep Text Classification Can be Fooled](https://mlsanity.com/paper/1704.08006/) (2017)
- [Crafting Adversarial Input Sequences for Recurrent Neural Networks](https://mlsanity.com/paper/1604.08275/) (2016)
- [Entity-Relation Extraction as Multi-Turn Question Answering](https://mlsanity.com/paper/1905.05529/) (2019)
- [Coreference Resolution as Query-based Span Prediction](https://mlsanity.com/paper/1911.01746/) (2019)
- [Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models](https://mlsanity.com/paper/2103.06922/) (2021)

## Built on by (most cited first, 4 of 4)

- [Pay Attention to Real World Perturbations! Natural Robustness Evaluation in Machine Reading Comprehension](https://mlsanity.com/paper/2502.16523/) (2025)
- [Single-Sentence Reader: A Novel Approach for Addressing Answer Position Bias](https://mlsanity.com/paper/2308.04566/) (2023)
- [AGent: A Novel Pipeline for Automatically Creating Unanswerable Questions](https://mlsanity.com/paper/2309.05103/) (2023)
- [Towards Robust Extractive Question Answering Models: Rethinking the Training Methodology](https://mlsanity.com/paper/2409.19766/) (2024)

## Abstract (arXiv)

Pretrained language models have achieved super-human performances on many Machine Reading Comprehension (MRC) benchmarks. Nevertheless, their relative inability to defend against adversarial attacks has spurred skepticism about their natural language understanding. In this paper, we ask whether training with unanswerable questions in SQuAD 2.0 can help improve the robustness of MRC models against adversarial attacks. To explore that question, we fine-tune three state-of-the-art language models on either SQuAD 1.1 or SQuAD 2.0 and then evaluate their robustness under adversarial attacks. Our experiments reveal that current models fine-tuned on SQuAD 2.0 do not initially appear to be any more robust than ones fine-tuned on SQuAD 1.1, yet they reveal a measure of hidden robustness that can be leveraged to realize actual performance gains. Furthermore, we find that the robustness of models fine-tuned on SQuAD 2.0 extends to additional out-of-domain datasets. Finally, we introduce a new adversarial attack to reveal artifacts of SQuAD 2.0 that current MRC models are learning.