# SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders

arXiv: https://arxiv.org/abs/1910.00998 · HTML: https://mlsanity.com/paper/1910.00998/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2019.

## Built on by (most cited first, 7 of 7)

- [Civil Rephrases Of Toxic Texts With Self-Supervised Transformers](https://mlsanity.com/paper/2102.05456/) (2021)
- [AgreeSum: Agreement-Oriented Multi-Document Summarization](https://mlsanity.com/paper/2106.02278/) (2021)
- [JaCoText: A Pretrained Model for Java Code-Text Generation](https://mlsanity.com/paper/2303.12869/) (2023)
- [Effective Pre-Training Objectives for Transformer-based Autoencoders](https://mlsanity.com/paper/2210.13536/) (2022)
- [Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation](https://mlsanity.com/paper/2110.07002/) (2021)
- [Unsupervised Summarization Re-ranking](https://mlsanity.com/paper/2212.09593/) (2022)
- [Structural Self-Supervised Objectives for Transformers](https://mlsanity.com/paper/2309.08272/) (2023)

## Abstract (arXiv)

We propose an end-to-end neural model for zero-shot abstractive text summarization of paragraphs, and introduce a benchmark task, ROCSumm, based on ROCStories, a subset for which we collected human summaries. In this task, five-sentence stories (paragraphs) are summarized with one sentence, using human summaries only for evaluation. We show results for extractive and human baselines to demonstrate a large abstractive gap in performance. Our model, SummAE, consists of a denoising auto-encoder that embeds sentences and paragraphs in a common space, from which either can be decoded. Summaries for paragraphs are generated by decoding a sentence from the paragraph representations. We find that traditional sequence-to-sequence auto-encoders fail to produce good summaries and describe how specific architectural choices and pre-training techniques can significantly improve performance, outperforming extractive baselines. The data, training, evaluation code, and best model weights are open-sourced.