# Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering

arXiv: https://arxiv.org/abs/2109.07009 · HTML: https://mlsanity.com/paper/2109.07009/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 70 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020)
- [Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text](https://mlsanity.com/paper/1809.00782/) (2018)
- [End-to-End Open-Domain Question Answering with BERTserini](https://mlsanity.com/paper/1902.01718/) (2019)
- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://mlsanity.com/paper/1911.10470/) (2019)
- [TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection](https://mlsanity.com/paper/1911.04118/) (2019)
- [What Makes Reading Comprehension Questions Easier?](https://mlsanity.com/paper/1808.09384/) (2018)
- [Answering Complex Open-domain Questions Through Iterative Query Generation](https://mlsanity.com/paper/1910.07000/) (2019)
- [Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering](https://mlsanity.com/paper/1911.03868/) (2019)
- [Knowing More About Questions Can Help: Improving Calibration in Question Answering](https://mlsanity.com/paper/2106.01494/) (2021)
- [Ask the Right Questions: Active Question Reformulation with Reinforcement Learning](https://mlsanity.com/paper/1705.07830/) (2017)
- [Improving Question Answering over Incomplete KBs with Knowledge-Aware Reader](https://mlsanity.com/paper/1905.07098/) (2019)
- [Quizbowl: The Case for Incremental Question Answering](https://mlsanity.com/paper/1904.04792/) (2019)
- [Identifying Well-formed Natural Language Questions](https://mlsanity.com/paper/1808.09419/) (2018)
- [A Compare-Aggregate Model for Matching Text Sequences](https://mlsanity.com/paper/1611.01747/) (2016)
- [Answering Complex Questions by Joining Multi-Document Evidence with Quasi Knowledge Graphs](https://mlsanity.com/paper/1908.00469/) (2019)
- [The Cascade Transformer: an Application for Efficient Answer Sentence Selection](https://mlsanity.com/paper/2005.02534/) (2020)
- [A Compare-Aggregate Model with Latent Clustering for Answer Selection](https://mlsanity.com/paper/1905.12897/) (2019)

## Built on by (most cited first, 19 of 19)

- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings](https://mlsanity.com/paper/2203.00211/) (2022)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [Learning to Reject with a Fixed Predictor: Application to Decontextualization](https://mlsanity.com/paper/2301.09044/) (2023)
- [Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA](https://mlsanity.com/paper/2305.01812/) (2023)
- [MetaQA: Combining Expert Agents for Multi-Skill Question Answering](https://mlsanity.com/paper/2112.01922/) (2021)
- [Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?](https://mlsanity.com/paper/2211.12707/) (2022)
- [Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection](https://mlsanity.com/paper/2205.10455/) (2022)
- [Paragraph-based Transformer Pre-training for Multi-Sentence Inference](https://mlsanity.com/paper/2205.01228/) (2022)
- [Knowledge Transfer from Answer Ranking to Answer Generation](https://mlsanity.com/paper/2210.12865/) (2022)
- [Learning Answer Generation using Supervision from Automatic Question Answering Evaluators](https://mlsanity.com/paper/2305.15344/) (2023)
- [In Situ Answer Sentence Selection at Web-scale](https://mlsanity.com/paper/2201.05984/) (2022)
- [Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages](https://mlsanity.com/paper/2305.16302/) (2023)
- [A Major Obstacle for NLP Research: Let's Talk about Time Allocation!](https://mlsanity.com/paper/2211.16858/) (2022)
- [Context-Aware Transformer Pre-Training for Answer Sentence Selection](https://mlsanity.com/paper/2305.15358/) (2023)
- [The Craft of Selective Prediction: Towards Reliable Case Outcome Classification -- An Empirical Study on European Court of Human Rights Cases](https://mlsanity.com/paper/2409.18645/) (2024)
- [Joint Training for Selective Prediction](https://mlsanity.com/paper/2410.24029/) (2024)
- [Investigating the Failure Modes of the AUC metric and Exploring Alternatives for Evaluating Systems in Safety Critical Applications](https://mlsanity.com/paper/2210.04466/) (2022)
- [SINF: Semantic Neural Network Inference with Semantic Subgraphs](https://mlsanity.com/paper/2310.01259/) (2023)

## Abstract (arXiv)

In this paper we propose a novel approach towards improving the efficiency of Question Answering (QA) systems by filtering out questions that will not be answered by them. This is based on an interesting new finding: the answer confidence scores of state-of-the-art QA systems can be approximated well by models solely using the input question text. This enables preemptive filtering of questions that are not answered by the system due to their answer confidence scores being lower than the system threshold. Specifically, we learn Transformer-based question models by distilling Transformer-based answering models. Our experiments on three popular QA datasets and one industrial QA benchmark demonstrate the ability of our question models to approximate the Precision/Recall curves of the target QA system well. These question models, when used as filters, can effectively trade off lower computation cost of QA systems for lower Recall, e.g., reducing computation by ~60%, while only losing ~3-4% of Recall.