# Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2311.17514 · HTML: https://mlsanity.com/paper/2311.17514/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 59 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [Learning Distributed Representations of Sentences from Unlabelled Data](https://mlsanity.com/paper/1602.03483/) (2016)
- [Extractive Opinion Summarization in Quantized Transformer Spaces](https://mlsanity.com/paper/2012.04443/) (2020)
- [Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised](https://mlsanity.com/paper/1808.08858/) (2018)
- [Multi-Reward Reinforced Summarization with Saliency and Entailment](https://mlsanity.com/paper/1804.06451/) (2018)
- [Unsupervised Opinion Summarization as Copycat-Review Generation](https://mlsanity.com/paper/1911.02247/) (2019)
- [PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval](https://mlsanity.com/paper/2108.06027/) (2021)
- [Scheduled Sampling for Transformers](https://mlsanity.com/paper/1906.07651/) (2019)
- [Aspect-Controllable Opinion Summarization](https://mlsanity.com/paper/2109.03171/) (2021)
- [Read before Generate! Faithful Long Form Question Answering with Machine Reading](https://mlsanity.com/paper/2203.00343/) (2022)
- [Diversity driven Attention Model for Query-based Abstractive Summarization](https://mlsanity.com/paper/1704.08300/) (2017)
- [Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching](https://mlsanity.com/paper/2004.12297/) (2020)
- [Generating Query Focused Summaries from Query-Free Resources](https://mlsanity.com/paper/2012.14774/) (2020)
- [The Summary Loop: Learning to Write Abstractive Summaries Without Examples](https://mlsanity.com/paper/2105.05361/) (2021)
- [Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization](https://mlsanity.com/paper/1909.00141/) (2019)
- [Differentiable Scheduled Sampling for Credit Assignment](https://mlsanity.com/paper/1704.06970/) (2017)
- [Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference](https://mlsanity.com/paper/2106.01186/) (2021)

## Built on by (most cited first, 1 of 1)

- [Query-Focused Event Summarization: A Dataset and Benchmark](https://mlsanity.com/paper/2607.11166/) (2026)

## Abstract (arXiv)

Query-focused Summarization (QfS) deals with systems that generate summaries from document(s) based on a query. Motivated by the insight that Reinforcement Learning (RL) provides a generalization to Supervised Learning (SL) for Natural Language Generation, and thereby performs better (empirically) than SL, we use an RL-based approach for this task of QfS. Additionally, we also resolve the conflict of employing RL in Transformers with Teacher Forcing. We develop multiple Policy Gradient networks, trained on various reward signals: ROUGE, BLEU, and Semantic Similarity, which lead to a 10-point improvement over the State-of-the-Art approach on the ROUGE-L metric for a benchmark dataset (ELI5). We also show performance of our approach in zero-shot setting for another benchmark dataset (DebatePedia) -- our approach leads to results comparable to baselines, which were specifically trained on DebatePedia. To aid the RL training, we propose a better semantic similarity reward, enabled by a novel Passage Embedding scheme developed using Cluster Hypothesis. Lastly, we contribute a gold-standard test dataset to further research in QfS and Long-form Question Answering (LfQA).