# RewardsOfSum: Exploring Reinforcement Learning Rewards for Summarisation

arXiv: https://arxiv.org/abs/2106.04080 · HTML: https://mlsanity.com/paper/2106.04080/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [Ranking Sentences for Extractive Summarization with Reinforcement Learning](https://mlsanity.com/paper/1802.08636/) (2018)
- [Multi-Reward Reinforced Summarization with Saliency and Entailment](https://mlsanity.com/paper/1804.06451/) (2018)
- [SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization](https://mlsanity.com/paper/2005.03724/) (2020)
- [A Simple, Fast Diverse Decoding Algorithm for Neural Generation](https://mlsanity.com/paper/1611.08562/) (2016)
- [Classical Structured Prediction Losses for Sequence to Sequence Learning](https://mlsanity.com/paper/1711.04956/) (2017)
- [Mutual Information and Diverse Decoding Improve Neural Machine Translation](https://mlsanity.com/paper/1601.00372/) (2016)
- [Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization](https://mlsanity.com/paper/1909.00141/) (2019)
- [Summary Level Training of Sentence Rewriting for Abstractive Summarization](https://mlsanity.com/paper/1909.08752/) (2019)
- [Continual BERT: Continual Learning for Adaptive Extractive Summarization of COVID-19 Literature](https://mlsanity.com/paper/2007.03405/) (2020)
- [Reinforced Extractive Summarization with Question-Focused Rewards](https://mlsanity.com/paper/1805.10392/) (2018)

## Built on by (most cited first, 4 of 4)

- [Human-in-the-loop Abstractive Dialogue Summarization](https://mlsanity.com/paper/2212.09750/) (2022)
- [Evaluating and Improving Factuality in Multimodal Abstractive Summarization](https://mlsanity.com/paper/2211.02580/) (2022)
- [A Multi-Document Coverage Reward for RELAXed Multi-Document Summarization](https://mlsanity.com/paper/2203.02894/) (2022)
- [Efficient and Interpretable Compressive Text Summarisation with Unsupervised Dual-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.03415/) (2023)

## Abstract (arXiv)

To date, most abstractive summarisation models have relied on variants of the negative log-likelihood (NLL) as their training objective. In some cases, reinforcement learning has been added to train the models with an objective that is closer to their evaluation measures (e.g. ROUGE). However, the reward function to be used within the reinforcement learning approach can play a key role for performance and is still partially unexplored. For this reason, in this paper, we propose two reward functions for the task of abstractive summarisation: the first function, referred to as RwB-Hinge, dynamically selects the samples for the gradient update. The second function, nicknamed RISK, leverages a small pool of strong candidates to inform the reward. In the experiments, we probe the proposed approach by fine-tuning an NLL pre trained model over nine summarisation datasets of diverse size and nature. The experimental results show a consistent improvement over the negative log-likelihood baselines.