# Towards Confident Machine Reading Comprehension

arXiv: https://arxiv.org/abs/2101.07942 · HTML: https://mlsanity.com/paper/2101.07942/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 20 Jan 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Retrospective Reader for Machine Reading Comprehension](https://mlsanity.com/paper/2001.09694/) (2020)
- [SG-Net: Syntax-Guided Machine Reading Comprehension](https://mlsanity.com/paper/1908.05147/) (2019)
- [Robust Machine Comprehension Models via Adversarial Training](https://mlsanity.com/paper/1804.06473/) (2018)
- [Out-of-Domain Detection for Low-Resource Text Classification Tasks](https://mlsanity.com/paper/1909.05357/) (2019)
- [A BERT Baseline for the Natural Questions](https://mlsanity.com/paper/1901.08634/) (2019)
- [Read + Verify: Machine Reading Comprehension with Unanswerable Questions](https://mlsanity.com/paper/1808.05759/) (2018)
- [Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering](https://mlsanity.com/paper/1711.05116/) (2017)
- [Automatically Learning Data Augmentation Policies for Dialogue Tasks](https://mlsanity.com/paper/1909.12868/) (2019)
- [RankQA: Neural Question Answering with Answer Re-Ranking](https://mlsanity.com/paper/1906.03008/) (2019)
- [Frustratingly Easy Natural Question Answering](https://mlsanity.com/paper/1909.05286/) (2019)

## Built on by (most cited first, 3 of 3)

- [Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering](https://mlsanity.com/paper/2109.07009/) (2021)
- [SmartBook: AI-Assisted Situation Report Generation for Intelligence Analysts](https://mlsanity.com/paper/2303.14337/) (2023)
- [Trusting Language Models in Education](https://mlsanity.com/paper/2308.03866/) (2023)

## Abstract (arXiv)

There has been considerable progress on academic benchmarks for the Reading Comprehension (RC) task with State-of-the-Art models closing the gap with human performance on extractive question answering. Datasets such as SQuAD 2.0 & NQ have also introduced an auxiliary task requiring models to predict when a question has no answer in the text. However, in production settings, it is also necessary to provide confidence estimates for the performance of the underlying RC model at both answer extraction and "answerability" detection. We propose a novel post-prediction confidence estimation model, which we call Mr.C (short for Mr. Confident), that can be trained to improve a system's ability to refrain from making incorrect predictions with improvements of up to 4 points as measured by Area Under the Curve (AUC) scores. Mr.C can benefit from a novel white-box feature that leverages the underlying RC model's gradients. Performance prediction is particularly important in cases of domain shift (as measured by training RC models on SQUAD 2.0 and evaluating on NQ), where Mr.C not only improves AUC, but also traditional answerability prediction (as measured by a 5 point improvement in F1).