# A Lightweight Constrained Generation Alternative for Query-focused Summarization

arXiv: https://arxiv.org/abs/2304.11721 · HTML: https://mlsanity.com/paper/2304.11721/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 23 Apr 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics](https://mlsanity.com/paper/2112.08726/) (2021)
- [NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints](https://mlsanity.com/paper/2010.12884/) (2020)
- [Pathologies of Neural Models Make Interpretations Difficult](https://mlsanity.com/paper/1804.07781/) (2018)
- [Explaining NLP Models via Minimal Contrastive Editing (MiCE)](https://mlsanity.com/paper/2012.13985/) (2020)
- [Guided Open Vocabulary Image Captioning with Constrained Beam Search](https://mlsanity.com/paper/1612.00576/) (2016)
- [Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline](https://mlsanity.com/paper/2101.08751/) (2021)
- [Diversity driven Attention Model for Query-based Abstractive Summarization](https://mlsanity.com/paper/1704.08300/) (2017)
- [Gradient-based Analysis of NLP Models is Manipulable](https://mlsanity.com/paper/2010.05419/) (2020)
- [An In-depth Investigation of User Response Simulation for Conversational Search](https://mlsanity.com/paper/2304.07944/) (2023)
- [Improve Query Focused Abstractive Summarization by Incorporating Answer Relevance](https://mlsanity.com/paper/2105.12969/) (2021)
- [Abstractive Snippet Generation](https://mlsanity.com/paper/2002.10782/) (2020)
- [Multi-hop Inference for Question-driven Summarization](https://mlsanity.com/paper/2010.03738/) (2020)
- [Conditional Self-Attention for Query-based Summarization](https://mlsanity.com/paper/2002.07338/) (2020)
- [Positional Bias in Long-Document Ranking: Impact, Assessment, and Mitigation](https://mlsanity.com/paper/2207.01262/) (2022)

## Built on by (most cited first, 8 of 8)

- [AI Hallucinations: A Misnomer Worth Clarifying](https://mlsanity.com/paper/2401.06796/) (2024)
- [Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression](https://mlsanity.com/paper/2407.04965/) (2024)
- [RankMamba: Benchmarking Mamba's Document Ranking Performance in the Era of Transformers](https://mlsanity.com/paper/2403.18276/) (2024)
- [Context-aware Decoding Reduces Hallucination in Query-focused Summarization](https://mlsanity.com/paper/2312.14335/) (2023)
- [State Space Models are Strong Text Rerankers](https://mlsanity.com/paper/2412.14354/) (2024)
- [CSPLADE: Learned Sparse Retrieval with Causal Language Models](https://mlsanity.com/paper/2504.10816/) (2025)
- [Distillation versus Contrastive Learning: How to Train Your Rerankers](https://mlsanity.com/paper/2507.08336/) (2025)
- [Leveraging Generative Models for Real-Time Query-Driven Text Summarization in Large-Scale Web Search](https://mlsanity.com/paper/2508.20559/) (2025)

## Abstract (arXiv)

Query-focused summarization (QFS) aims to provide a summary of a document that satisfies information need of a given query and is useful in various IR applications, such as abstractive snippet generation. Current QFS approaches typically involve injecting additional information, e.g. query-answer relevance or fine-grained token-level interaction between a query and document, into a finetuned large language model. However, these approaches often require extra parameters & training, and generalize poorly to new dataset distributions. To mitigate this, we propose leveraging a recently developed constrained generation model Neurological Decoding (NLD) as an alternative to current QFS regimes which rely on additional sub-architectures and training. We first construct lexical constraints by identifying important tokens from the document using a lightweight gradient attribution model, then subsequently force the generated summary to satisfy these constraints by directly manipulating the final vocabulary likelihood. This lightweight approach requires no additional parameters or finetuning as it utilizes both an off-the-shelf neural retrieval model to construct the constraints and a standard generative language model to produce the QFS. We demonstrate the efficacy of this approach on two public QFS collections achieving near parity with the state-of-the-art model with substantially reduced complexity.