# Conditional Self-Attention for Query-based Summarization

arXiv: https://arxiv.org/abs/2002.07338 · HTML: https://mlsanity.com/paper/2002.07338/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2020.

## Built on by (most cited first, 6 of 6)

- [Exploring Neural Models for Query-Focused Summarization](https://mlsanity.com/paper/2112.07637/) (2021)
- [Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization](https://mlsanity.com/paper/2112.11670/) (2021)
- [Improve Query Focused Abstractive Summarization by Incorporating Answer Relevance](https://mlsanity.com/paper/2105.12969/) (2021)
- [A Lightweight Constrained Generation Alternative for Query-focused Summarization](https://mlsanity.com/paper/2304.11721/) (2023)
- [Generative Long-form Question Answering: Relevance, Faithfulness and Succinctness](https://mlsanity.com/paper/2211.08386/) (2022)
- [Does This Summary Answer My Question? Modeling Query-Focused Summary Readers with Rational Speech Acts](https://mlsanity.com/paper/2411.06524/) (2024)

## Abstract (arXiv)

Self-attention mechanisms have achieved great success on a variety of NLP tasks due to its flexibility of capturing dependency between arbitrary positions in a sequence. For problems such as query-based summarization (Qsumm) and knowledge graph reasoning where each input sequence is associated with an extra query, explicitly modeling such conditional contextual dependencies can lead to a more accurate solution, which however cannot be captured by existing self-attention mechanisms. In this paper, we propose textitconditional self-attention (CSA), a neural network module designed for conditional dependency modeling. CSA works by adjusting the pairwise attention between input tokens in a self-attention module with the matching score of the inputs to the given query. Thereby, the contextual dependencies modeled by CSA will be highly relevant to the query. We further studied variants of CSA defined by different types of attention. Experiments on Debatepedia and HotpotQA benchmark datasets show CSA consistently outperforms vanilla Transformer and previous models for the Qsumm problem.