# Can NLI Models Verify QA Systems' Predictions?

arXiv: https://arxiv.org/abs/2104.08731 · HTML: https://mlsanity.com/paper/2104.08731/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 67 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 30 of 43)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [Dialogue Natural Language Inference](https://mlsanity.com/paper/1811.00671/) (2018)
- [Combining Fact Extraction and Verification with Neural Semantic Matching Networks](https://mlsanity.com/paper/1811.07039/) (2018)
- [Decontextualization: Making Sentences Stand-Alone](https://mlsanity.com/paper/2102.05169/) (2021)
- [Transforming Question Answering Datasets Into Natural Language Inference Datasets](https://mlsanity.com/paper/1809.02922/) (2018)
- [Understanding Dataset Design Choices for Multi-hop Reasoning](https://mlsanity.com/paper/1904.12106/) (2019)

## Built on by (most cited first, 30 of 46)

- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [WiCE: Real-World Entailment for Claims in Wikipedia](https://mlsanity.com/paper/2303.01432/) (2023)
- [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://mlsanity.com/paper/2202.07654/) (2022)
- [Generating Literal and Implied Subquestions to Fact-check Complex Claims](https://mlsanity.com/paper/2205.06938/) (2022)
- [(QA)2: Question Answering with Questionable Assumptions](https://mlsanity.com/paper/2212.10003/) (2022)
- [Semantic Answer Similarity for Evaluating Question Answering Models](https://mlsanity.com/paper/2108.06130/) (2021)
- [Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters](https://mlsanity.com/paper/2204.07447/) (2022)
- [Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements](https://mlsanity.com/paper/2305.03695/) (2023)
- [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2112.08688/) (2021)
- [Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference](https://mlsanity.com/paper/2211.11875/) (2022)
- [FaVIQ: FAct Verification from Information-seeking Questions](https://mlsanity.com/paper/2107.02153/) (2021)
- [Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing](https://mlsanity.com/paper/2305.16635/) (2023)
- [REV: Information-Theoretic Evaluation of Free-Text Rationales](https://mlsanity.com/paper/2210.04982/) (2022)
- [Mitigating Temporal Misalignment by Discarding Outdated Facts](https://mlsanity.com/paper/2305.14824/) (2023)
- [AmbigDocs: Reasoning across Documents on Different Entities under the Same Name](https://mlsanity.com/paper/2404.12447/) (2024)
- [Conformal Linguistic Calibration: Trading-off between Factuality and Specificity](https://mlsanity.com/paper/2502.19110/) (2025)
- [Ultra-fine Entity Typing with Indirect Supervision from Natural Language Inference](https://mlsanity.com/paper/2202.06167/) (2022)
- [Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses](https://mlsanity.com/paper/2305.11662/) (2023)
- [Selective Generation for Controllable Language Models](https://mlsanity.com/paper/2307.09254/) (2023)
- [Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Facts](https://mlsanity.com/paper/2405.06524/) (2024)
- [Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification](https://mlsanity.com/paper/2402.03686/) (2024)
- [Accurate and Nuanced Open-QA Evaluation Through Textual Entailment](https://mlsanity.com/paper/2405.16702/) (2024)
- [ZARA: Improving Few-Shot Self-Rationalization for Small Language Models](https://mlsanity.com/paper/2305.07355/) (2023)
- [Concise Answers to Complex Questions: Summarization of Long-form Answers](https://mlsanity.com/paper/2305.19271/) (2023)
- [From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency](https://mlsanity.com/paper/2404.12145/) (2024)
- [NarraSum: A Large-Scale Dataset for Abstractive Narrative Summarization](https://mlsanity.com/paper/2212.01476/) (2022)
- [Explanation-aware Soft Ensemble Empowers Large Language Model In-context Learning](https://mlsanity.com/paper/2311.07099/) (2023)
- [RORA: Robust Free-Text Rationale Evaluation](https://mlsanity.com/paper/2402.18678/) (2024)

## Abstract (arXiv)

To build robust question answering systems, we need the ability to verify whether answers to questions are truly correct, not just "good enough" in the context of imperfect QA datasets. We explore the use of natural language inference (NLI) as a way to achieve this goal, as NLI inherently requires the premise (document context) to contain all necessary information to support the hypothesis (proposed answer to the question). We leverage large pre-trained models and recent prior datasets to construct powerful question converter and decontextualization modules, which can reformulate QA instances as premise-hypothesis pairs with very high reliability. Then, by combining standard NLI datasets with NLI examples automatically derived from QA training data, we can train NLI models to judge the correctness of QA models' proposed answers. We show that our NLI approach can generally improve the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. Careful manual analysis over the predictions of our NLI model shows that it can further identify cases where the QA model produces the right answer for the wrong reason, or where the answer cannot be verified as addressing all aspects of the question.