# On Extractive and Abstractive Neural Document Summarization with Transformer Language Models

arXiv: https://arxiv.org/abs/1909.03186 · HTML: https://mlsanity.com/paper/1909.03186/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2019.

## Built on by (most cited first, 30 of 83)

- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://mlsanity.com/paper/2105.06762/) (2021)
- [In-Context Impersonation Reveals Large Language Models' Strengths and Biases](https://mlsanity.com/paper/2305.14930/) (2023)
- [Lift Yourself Up: Retrieval-augmented Text Generation with Self Memory](https://mlsanity.com/paper/2305.02437/) (2023)
- [Understanding the Benefits and Challenges of Using Large Language Model-based Conversational Agents for Mental Well-being Support](https://mlsanity.com/paper/2307.15810/) (2023)
- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents](https://mlsanity.com/paper/2110.10150/) (2021)
- [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021)
- [Poolingformer: Long Document Modeling with Pooling Attention](https://mlsanity.com/paper/2105.04371/) (2021)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [A Survey on Long Text Modeling with Transformers](https://mlsanity.com/paper/2302.14502/) (2023)
- [Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder](https://mlsanity.com/paper/2304.04052/) (2023)
- [Long Document Summarization with Top-down and Bottom-up Inference](https://mlsanity.com/paper/2203.07586/) (2022)
- [Submodularity In Machine Learning and Artificial Intelligence](https://mlsanity.com/paper/2202.00132/) (2022)
- [Bringing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific Documents](https://mlsanity.com/paper/2106.00130/) (2021)
- [HEGEL: Hypergraph Transformer for Long Document Summarization](https://mlsanity.com/paper/2210.04126/) (2022)
- [Toward Unifying Text Segmentation and Long Document Summarization](https://mlsanity.com/paper/2210.16422/) (2022)
- [HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Information](https://mlsanity.com/paper/2203.09629/) (2022)
- [How Far are We from Robust Long Abstractive Summarization?](https://mlsanity.com/paper/2210.16732/) (2022)
- [Long Document Summarization in a Low Resource Setting using Pretrained Language Models](https://mlsanity.com/paper/2103.00751/) (2021)
- [Long-Span Summarization via Local Attention and Content Selection](https://mlsanity.com/paper/2105.03801/) (2021)
- [OASum: Large-Scale Open Domain Aspect-based Summarization](https://mlsanity.com/paper/2212.09233/) (2022)
- [Topic-Aware Contrastive Learning for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2109.04994/) (2021)
- [BASS: Boosting Abstractive Summarization with Unified Semantic Graph](https://mlsanity.com/paper/2105.12041/) (2021)
- [Automated scholarly paper review: Concepts, technologies, and challenges](https://mlsanity.com/paper/2111.07533/) (2021)
- [Enriching and Controlling Global Semantics for Text Summarization](https://mlsanity.com/paper/2109.10616/) (2021)
- [ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications](https://mlsanity.com/paper/2403.05303/) (2024)
- [MemSum: Extractive Summarization of Long Documents Using Multi-Step Episodic Markov Decision Processes](https://mlsanity.com/paper/2107.08929/) (2021)
- [Word Graph Guided Summarization for Radiology Findings](https://mlsanity.com/paper/2112.09925/) (2021)

## Abstract (arXiv)

We present a method to produce abstractive summaries of long documents that exceed several thousand words via neural abstractive summarization. We perform a simple extractive step before generating a summary, which is then used to condition the transformer language model on relevant information before being tasked with generating a summary. We show that this extractive step significantly improves summarization results. We also show that this approach produces more abstractive summaries compared to prior work that employs a copy mechanism while still achieving higher rouge scores. Note: The abstract above was not written by the authors, it was generated by one of the models presented in this paper.