# Choose Your QA Model Wisely: A Systematic Study of Generative and Extractive Readers for Question Answering arXiv: https://arxiv.org/abs/2203.07522 ยท HTML: https://mlsanity.com/paper/2203.07522/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 24 of its 32 references that are in the map. - First posted on arXiv 14 Mar 2022. ## Builds on (most cited first, 24 of 24) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016) - [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020) - [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014) - [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020) - [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021) - [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018) - [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020) - [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018) - [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016) - [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019) - [Reframing Instructional Prompts to GPTk's Language](https://mlsanity.com/paper/2109.07830/) (2021) - [Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval](https://mlsanity.com/paper/2009.12756/) (2020) - [UnitedQA: A Hybrid Approach for Open Domain Question Answering](https://mlsanity.com/paper/2101.00178/) (2021) - [General-Purpose Question-Answering with Macaw](https://mlsanity.com/paper/2109.02593/) (2021) - [Improving Biomedical Information Retrieval with Neural Retrievers](https://mlsanity.com/paper/2201.07745/) (2022) - [Our Evaluation Metric Needs an Update to Encourage Generalization](https://mlsanity.com/paper/2007.06898/) (2020) - [Pruning the Index Contents for Memory Efficient Open-Domain QA](https://mlsanity.com/paper/2102.10697/) (2021) ## Built on by (most cited first, 11 of 11) - [Is a Question Decomposition Unit All We Need?](https://mlsanity.com/paper/2205.12538/) (2022) - [Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models](https://mlsanity.com/paper/2310.00836/) (2023) - [In-BoXBART: Get Instructions into Biomedical Multi-Task Learning](https://mlsanity.com/paper/2204.07600/) (2022) - [BioTABQA: Instruction Learning for Biomedical Table Question Answering](https://mlsanity.com/paper/2207.02419/) (2022) - [Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?](https://mlsanity.com/paper/2211.12707/) (2022) - [Knowledge AI: Fine-tuning NLP Models for Facilitating Scientific Knowledge Extraction and Understanding](https://mlsanity.com/paper/2408.04651/) (2024) - [Neural Retriever and Go Beyond: A Thesis Proposal](https://mlsanity.com/paper/2205.16005/) (2022) - [Towards Efficient Methods in Medical Question Answering using Knowledge Graph Embeddings](https://mlsanity.com/paper/2401.07977/) (2024) - [OMoS-QA: A Dataset for Cross-Lingual Extractive Question Answering in a German Migration Context](https://mlsanity.com/paper/2407.15736/) (2024) - [Investigating Consistency in Query-Based Meeting Summarization: A Comparative Study of Different Embedding Methods](https://mlsanity.com/paper/2402.06907/) (2024) - [Context Convergence Improves Answering Inferential Questions](https://mlsanity.com/paper/2605.12370/) (2026) ## Abstract (arXiv) While both extractive and generative readers have been successfully applied to the Question Answering (QA) task, little attention has been paid toward the systematic comparison of them. Characterizing the strengths and weaknesses of the two readers is crucial not only for making a more informed reader selection in practice but also for developing a deeper understanding to foster further research on improving readers in a principled manner. Motivated by this goal, we make the first attempt to systematically study the comparison of extractive and generative readers for question answering. To be aligned with the state-of-the-art, we explore nine transformer-based large pre-trained language models (PrLMs) as backbone architectures. Furthermore, we organize our findings under two main categories: (1) keeping the architecture invariant, and (2) varying the underlying PrLMs. Among several interesting findings, it is important to highlight that (1) the generative readers perform better in long context QA, (2) the extractive readers perform better in short context while also showing better out-of-domain generalization, and (3) the encoder of encoder-decoder PrLMs (e.g., T5) turns out to be a strong extractive reader and outperforms the standard choice of encoder-only PrLMs (e.g., RoBERTa). We also study the effect of multi-task learning on the two types of readers varying the underlying PrLMs and perform qualitative and quantitative diagnosis to provide further insights into future directions in modeling better readers.