# DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization

arXiv: https://arxiv.org/abs/2110.08168 · HTML: https://mlsanity.com/paper/2110.08168/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 49 references that are in the map.
- First posted on arXiv 15 Oct 2021.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization](https://mlsanity.com/paper/1906.03741/) (2019)
- [Ranking Sentences for Extractive Summarization with Reinforcement Learning](https://mlsanity.com/paper/1802.08636/) (2018)
- [BillSum: A Corpus for Automatic Summarization of US Legislation](https://mlsanity.com/paper/1910.00523/) (2019)
- [MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://mlsanity.com/paper/2103.06410/) (2021)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [Extractive Summarization of Long Documents by Combining Global and Local Context](https://mlsanity.com/paper/1909.08089/) (2019)
- [Pretraining-Based Natural Language Generation for Text Summarization](https://mlsanity.com/paper/1902.09243/) (2019)
- [Neural Extractive Text Summarization with Syntactic Compression](https://mlsanity.com/paper/1902.00863/) (2019)
- [Scoring Sentence Singletons and Pairs for Abstractive Summarization](https://mlsanity.com/paper/1906.00077/) (2019)
- [Hierarchical Learning for Generation with Long Source Sequences](https://mlsanity.com/paper/2104.07545/) (2021)

## Built on by (most cited first, 30 of 34)

- [Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents](https://mlsanity.com/paper/2110.10150/) (2021)
- [A Survey on Long Text Modeling with Transformers](https://mlsanity.com/paper/2302.14502/) (2023)
- [Exploring Neural Models for Query-Focused Summarization](https://mlsanity.com/paper/2112.07637/) (2021)
- [Toward Unifying Text Segmentation and Long Document Summarization](https://mlsanity.com/paper/2210.16422/) (2022)
- [How Far are We from Robust Long Abstractive Summarization?](https://mlsanity.com/paper/2210.16732/) (2022)
- [Long Text and Multi-Table Summarization: Dataset and Method](https://mlsanity.com/paper/2302.03815/) (2023)
- [Adapting Pretrained Text-to-Text Models for Long Text Sequences](https://mlsanity.com/paper/2209.10052/) (2022)
- [ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications](https://mlsanity.com/paper/2403.05303/) (2024)
- [SNaC: Coherence Error Detection for Narrative Summarization](https://mlsanity.com/paper/2205.09641/) (2022)
- [VCSUM: A Versatile Chinese Meeting Summarization Dataset](https://mlsanity.com/paper/2305.05280/) (2023)
- [ReMamba: Equip Mamba with Effective Long-Sequence Modeling](https://mlsanity.com/paper/2408.15496/) (2024)
- [Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsets](https://mlsanity.com/paper/2310.13340/) (2023)
- [Leveraging Locality in Abstractive Text Summarization](https://mlsanity.com/paper/2205.12476/) (2022)
- [Generating EDU Extracts for Plan-Guided Summary Re-Ranking](https://mlsanity.com/paper/2305.17779/) (2023)
- [Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers](https://mlsanity.com/paper/2308.13191/) (2023)
- [Factorizing Content and Budget Decisions in Abstractive Summarization of Long Documents](https://mlsanity.com/paper/2205.12486/) (2022)
- [Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Study](https://mlsanity.com/paper/2306.01169/) (2023)
- [Learning to Rank Utterances for Query-Focused Meeting Summarization](https://mlsanity.com/paper/2305.12753/) (2023)
- [Query-Utterance Attention with Joint modeling for Query-Focused Meeting Summarization](https://mlsanity.com/paper/2303.04487/) (2023)
- [Unsupervised Abstractive Dialogue Summarization with Word Graphs and POV Conversion](https://mlsanity.com/paper/2205.13108/) (2022)
- [AWESOME: GPU Memory-constrained Long Document Summarization using Memory Mechanism and Global Salient Content](https://mlsanity.com/paper/2305.14806/) (2023)
- [Label-Free Topic-Focused Summarization Using Query Augmentation](https://mlsanity.com/paper/2404.16411/) (2024)
- [A Mixed-Language Multi-Document News Summarization Dataset and a Graphs-Based Extract-Generate Model](https://mlsanity.com/paper/2410.09773/) (2024)
- [Adaptive Planning for Multi-Attribute Controllable Summarization with Monte Carlo Tree Search](https://mlsanity.com/paper/2509.26435/) (2025)
- [Improving Query-Focused Meeting Summarization with Query-Relevant Knowledge](https://mlsanity.com/paper/2309.02105/) (2023)
- [Exploring Iterative Controllable Summarization with Large Language Models](https://mlsanity.com/paper/2411.12460/) (2024)
- [R2F: A General Retrieval, Reading and Fusion Framework for Document-level Natural Language Inference](https://mlsanity.com/paper/2210.12328/) (2022)
- [Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences](https://mlsanity.com/paper/2212.09726/) (2022)
- [Summarization of Investment Reports Using Pre-trained Model](https://mlsanity.com/paper/2408.01744/) (2024)
- [A Unified Retrieval Framework with Document Ranking and EDU Filtering for Multi-document Summarization](https://mlsanity.com/paper/2504.16711/) (2025)

## Abstract (arXiv)

Transformer-based models have achieved state-of-the-art performance on short-input summarization. However, they still struggle with summarizing longer text. In this paper, we present DYLE, a novel dynamic latent extraction approach for abstractive long-input summarization. DYLE jointly trains an extractor and a generator and treats the extracted text snippets as the latent variable, allowing dynamic snippet-level attention weights during decoding. To provide adequate supervision, we propose simple yet effective heuristics for oracle extraction as well as a consistency loss term, which encourages the extractor to approximate the averaged dynamic weights predicted by the generator. We evaluate our method on different long-document and long-dialogue summarization tasks: GovReport, QMSum, and arXiv. Experiment results show that DYLE outperforms all existing methods on GovReport and QMSum, with gains up to 6.1 ROUGE, while yielding strong results on arXiv. Further analysis shows that the proposed dynamic weights provide interpretability of our generation process.