# Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension

arXiv: https://arxiv.org/abs/2404.17991 · HTML: https://mlsanity.com/paper/2404.17991/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 52 references that are in the map.
- First posted on arXiv 27 Apr 2024.

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [A Survey on Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2202.01110/) (2022)
- [Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering](https://mlsanity.com/paper/2104.08202/) (2021)
- [Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning](https://mlsanity.com/paper/1908.05803/) (2019)
- [Retrieve and Refine: Improved Sequence Generation Models For Dialogue](https://mlsanity.com/paper/1808.04776/) (2018)
- [Coreferential Reasoning Learning for Language Representation](https://mlsanity.com/paper/2004.06870/) (2020)
- [A Controllable Model of Grounded Response Generation](https://mlsanity.com/paper/2005.00613/) (2020)
- [Read before Generate! Faithful Long Form Question Answering with Machine Reading](https://mlsanity.com/paper/2203.00343/) (2022)
- [A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning](https://mlsanity.com/paper/1908.05514/) (2019)
- [Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning](https://mlsanity.com/paper/2306.03350/) (2023)
- [Question Directed Graph Attention Network for Numerical Reasoning over Text](https://mlsanity.com/paper/2009.07448/) (2020)
- [Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering](https://mlsanity.com/paper/1811.11934/) (2018)
- [RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generation](https://mlsanity.com/paper/2306.07206/) (2023)
- [Prototype-to-Style: Dialogue Generation with Style-Aware Editing on Retrieval Memory](https://mlsanity.com/paper/2004.02214/) (2020)
- [LIQUID: A Framework for List Question Answering Dataset Generation](https://mlsanity.com/paper/2302.01691/) (2023)
- [A Simple but Effective Method to Incorporate Multi-turn Context with BERT for Conversational Machine Comprehension](https://mlsanity.com/paper/1905.12848/) (2019)
- [Reinforcement Learning with Token-level Feedback for Controllable Text Generation](https://mlsanity.com/paper/2403.11558/) (2024)

## Abstract (arXiv)

Machine Reading Comprehension (MRC) poses a significant challenge in the field of Natural Language Processing (NLP). While mainstream MRC methods predominantly leverage extractive strategies using encoder-only models such as BERT, generative approaches face the issue of out-of-control generation -- a critical problem where answers generated are often incorrect, irrelevant, or unfaithful to the source text. To address these limitations in generative models for MRC, we introduce the Question-Attended Span Extraction (QASE) module. Integrated during the fine-tuning phase of pre-trained generative language models (PLMs), QASE significantly enhances their performance, allowing them to surpass the extractive capabilities of advanced Large Language Models (LLMs) such as GPT-4 in few-shot settings. Notably, these gains in performance do not come with an increase in computational demands. The efficacy of the QASE module has been rigorously tested across various datasets, consistently achieving or even surpassing state-of-the-art (SOTA) results, thereby bridging the gap between generative and extractive models in extractive MRC tasks.