# Exploring Neural Models for Query-Focused Summarization

arXiv: https://arxiv.org/abs/2112.07637 · HTML: https://mlsanity.com/paper/2112.07637/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 63 references that are in the map.
- First posted on arXiv 14 Dec 2021.

## Builds on (most cited first, 30 of 44)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Neural Text Summarization: A Critical Evaluation](https://mlsanity.com/paper/1908.08960/) (2019)
- [BookSum: A Collection of Datasets for Long-form Narrative Summarization](https://mlsanity.com/paper/2105.08209/) (2021)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization](https://mlsanity.com/paper/1906.03741/) (2019)
- [Hierarchical Transformers for Multi-Document Summarization](https://mlsanity.com/paper/1905.13164/) (2019)
- [SummScreen: A Dataset for Abstractive Screenplay Summarization](https://mlsanity.com/paper/2104.07091/) (2021)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents](https://mlsanity.com/paper/2110.10150/) (2021)
- [Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation](https://mlsanity.com/paper/2010.12836/) (2020)
- [AQuaMuSe: Automatically Generating Datasets for Query-Based Multi-Document Summarization](https://mlsanity.com/paper/2010.12694/) (2020)
- [Query Focused Abstractive Summarization: Incorporating Query Relevance, Multi-Document Coverage, and Summary Length Constraints into seq2seq Models](https://mlsanity.com/paper/1801.07704/) (2018)

## Built on by (most cited first, 30 of 44)

- [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://mlsanity.com/paper/2407.01370/) (2024)
- [Efficient Long-Text Understanding with Short-Text Models](https://mlsanity.com/paper/2208.00748/) (2022)
- [Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization](https://mlsanity.com/paper/2311.09184/) (2023)
- [SQuALITY: Building a Long-Document Summarization Dataset the Hard Way](https://mlsanity.com/paper/2205.11465/) (2022)
- [Learning to Plan and Generate Text with Citations](https://mlsanity.com/paper/2404.03381/) (2024)
- [OASum: Large-Scale Open Domain Aspect-based Summarization](https://mlsanity.com/paper/2212.09233/) (2022)
- [Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization](https://mlsanity.com/paper/2112.11670/) (2021)
- [Adapting Pretrained Text-to-Text Models for Long Text Sequences](https://mlsanity.com/paper/2209.10052/) (2022)
- [Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization](https://mlsanity.com/paper/2212.10449/) (2022)
- [SciReviewGen: A Large-scale Dataset for Automatic Literature Review Generation](https://mlsanity.com/paper/2305.15186/) (2023)
- [Hierarchical Catalogue Generation for Literature Review: A Benchmark](https://mlsanity.com/paper/2304.03512/) (2023)
- [Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression](https://mlsanity.com/paper/2402.16058/) (2024)
- [PDSum: Prototype-driven Continuous Summarization of Evolving Multi-document Sets Stream](https://mlsanity.com/paper/2302.05550/) (2023)
- [CQSumDP: A ChatGPT-Annotated Resource for Query-Focused Abstractive Summarization Based on Debatepedia](https://mlsanity.com/paper/2305.06147/) (2023)
- [Concise Answers to Complex Questions: Summarization of Long-form Answers](https://mlsanity.com/paper/2305.19271/) (2023)
- [Towards Diverse and Effective Question-Answer Pair Generation from Children Storybooks](https://mlsanity.com/paper/2306.06605/) (2023)
- [SumREN: Summarizing Reported Speech about Events in News](https://mlsanity.com/paper/2212.01146/) (2022)
- [Learning to Rank Utterances for Query-Focused Meeting Summarization](https://mlsanity.com/paper/2305.12753/) (2023)
- [QontSum: On Contrasting Salient Content for Query-focused Summarization](https://mlsanity.com/paper/2307.07586/) (2023)
- [On The Persona-based Summarization of Domain-Specific Documents](https://mlsanity.com/paper/2406.03986/) (2024)
- [GenerationPrograms: Fine-grained Attribution with Executable Programs](https://mlsanity.com/paper/2506.14580/) (2025)
- [OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviews](https://mlsanity.com/paper/2509.00285/) (2025)
- [Meeting Summarization: A Survey of the State of the Art](https://mlsanity.com/paper/2212.08206/) (2022)
- [Query-Utterance Attention with Joint modeling for Query-Focused Meeting Summarization](https://mlsanity.com/paper/2303.04487/) (2023)
- [Learning to Rank Salient Content for Query-focused Summarization](https://mlsanity.com/paper/2411.00324/) (2024)
- [StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs](https://mlsanity.com/paper/2412.18011/) (2024)
- [Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approaches](https://mlsanity.com/paper/2502.06617/) (2025)
- [Finding What Matters: Anchoring Context Knowledge with Evolving Indices for Iterative Retrieval](https://mlsanity.com/paper/2601.16462/) (2026)
- [Surveying the Landscape of Text Summarization with Deep Learning: A Comprehensive Review](https://mlsanity.com/paper/2310.09411/) (2023)
- [Instructive Dialogue Summarization with Query Aggregations](https://mlsanity.com/paper/2310.10981/) (2023)

## Abstract (arXiv)

Query-focused summarization (QFS) aims to produce summaries that answer particular questions of interest, enabling greater user control and personalization. While recently released datasets, such as QMSum or AQuaMuSe, facilitate research efforts in QFS, the field lacks a comprehensive study of the broad space of applicable modeling methods. In this paper we conduct a systematic exploration of neural approaches to QFS, considering two general classes of methods: two-stage extractive-abstractive solutions and end-to-end models. Within those categories, we investigate existing models and explore strategies for transfer learning. We also present two modeling extensions that achieve state-of-the-art performance on the QMSum dataset, up to a margin of 3.38 ROUGE-1, 3.72 ROUGE2, and 3.28 ROUGE-L when combined with transfer learning strategies. Results from human evaluation suggest that the best models produce more comprehensive and factually consistent summaries compared to a baseline model. Code and checkpoints are made publicly available: https://github.com/salesforce/query-focused-sum.