# Attention-guided Generative Models for Extractive Question Answering

arXiv: https://arxiv.org/abs/2110.06393 · HTML: https://mlsanity.com/paper/2110.06393/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 29 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [Distilling Knowledge from Reader to Retriever for Question Answering](https://mlsanity.com/paper/2012.04584/) (2020)
- [Detecting Hallucinated Content in Conditional Neural Sequence Generation](https://mlsanity.com/paper/2011.02593/) (2020)
- [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020)
- [Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering](https://mlsanity.com/paper/1908.08167/) (2019)
- [Relevance-guided Supervision for OpenQA with ColBERT](https://mlsanity.com/paper/2007.00814/) (2020)
- [Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering](https://mlsanity.com/paper/1911.03868/) (2019)

## Built on by (most cited first, 14 of 14)

- [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2112.08688/) (2021)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [Knowledge Transfer from Answer Ranking to Answer Generation](https://mlsanity.com/paper/2210.12865/) (2022)
- [Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models](https://mlsanity.com/paper/2211.00915/) (2022)
- [FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering](https://mlsanity.com/paper/2211.10147/) (2022)
- [Fine-tuning and aligning question answering models for complex information extraction tasks](https://mlsanity.com/paper/2309.14805/) (2023)
- [Who Are All The Stochastic Parrots Imitating? They Should Tell Us!](https://mlsanity.com/paper/2310.10583/) (2023)
- [Adapting Pre-trained Generative Models for Extractive Question Answering](https://mlsanity.com/paper/2311.02961/) (2023)
- [TextBox 2.0: A Text Generation Library with Pre-trained Language Models](https://mlsanity.com/paper/2212.13005/) (2022)
- [Learning Answer Generation using Supervision from Automatic Question Answering Evaluators](https://mlsanity.com/paper/2305.15344/) (2023)
- [Exploring Language Model Generalization in Low-Resource Extractive QA](https://mlsanity.com/paper/2409.18446/) (2024)
- [Detrimental Contexts in Open-Domain Question Answering](https://mlsanity.com/paper/2310.18077/) (2023)
- [Blending Reward Functions via Few Expert Demonstrations for Faithful and Accurate Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2311.00953/) (2023)
- [Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension](https://mlsanity.com/paper/2404.17991/) (2024)

## Abstract (arXiv)

We propose a novel method for applying Transformer models to extractive question answering (QA) tasks. Recently, pretrained generative sequence-to-sequence (seq2seq) models have achieved great success in question answering. Contributing to the success of these models are internal attention mechanisms such as cross-attention. We propose a simple strategy to obtain an extractive answer span from the generative model by leveraging the decoder cross-attention patterns. Viewing cross-attention as an architectural prior, we apply joint training to further improve QA performance. Empirical results show that on open-domain question answering datasets like NaturalQuestions and TriviaQA, our method approaches state-of-the-art performance on both generative and extractive inference, all while using much fewer parameters. Furthermore, this strategy allows us to perform hallucination-free inference while conferring significant improvements to the model's ability to rerank relevant passages.