# A Deep Reinforced Model for Abstractive Summarization

arXiv: https://arxiv.org/abs/1705.04304 · HTML: https://mlsanity.com/paper/1705.04304/

## Key facts

- Cited by 365 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2017.

## Built on by (most cited first, 30 of 365)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [SLiC-HF: Sequence Likelihood Calibration with Human Feedback](https://mlsanity.com/paper/2305.10425/) (2023)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL](https://mlsanity.com/paper/2402.19446/) (2024)
- [BookSum: A Collection of Datasets for Long-form Narrative Summarization](https://mlsanity.com/paper/2105.08209/) (2021)
- [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://mlsanity.com/paper/2105.06762/) (2021)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [The pitfalls of next-token prediction](https://mlsanity.com/paper/2403.06963/) (2024)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Offline RL for Natural Language Generation with Implicit Language Q Learning](https://mlsanity.com/paper/2206.11871/) (2022)
- [RL with KL penalties is better viewed as Bayesian inference](https://mlsanity.com/paper/2205.11275/) (2022)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [Language Models Learn to Mislead Humans via RLHF](https://mlsanity.com/paper/2409.12822/) (2024)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Calibrating Sequence likelihood Improves Conditional Language Generation](https://mlsanity.com/paper/2210.00045/) (2022)
- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [Aligning Language Models with Preferences through f-divergence Minimization](https://mlsanity.com/paper/2302.08215/) (2023)
- [f-Divergence Minimization for Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2307.15190/) (2023)

## Abstract (arXiv)

Attentional, RNN-based encoder-decoder models for abstractive summarization have achieved good performance on short input and output sequences. For longer documents and summaries however these models often include repetitive and incoherent phrases. We introduce a neural network model with a novel intra-attention that attends over the input and continuously generated output separately, and a new training method that combines standard supervised word prediction and reinforcement learning (RL). Models trained only with supervised learning often exhibit "exposure bias" - they assume ground truth is provided at each step during training. However, when standard word prediction is combined with the global sequence prediction training of RL the resulting summaries become more readable. We evaluate this model on the CNN/Daily Mail and New York Times datasets. Our model obtains a 41.16 ROUGE-1 score on the CNN/Daily Mail dataset, an improvement over previous state-of-the-art models. Human evaluation also shows that our model produces higher quality summaries.