# TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising

arXiv: https://arxiv.org/abs/2001.00725 · HTML: https://mlsanity.com/paper/2001.00725/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jan 2020.

## Built on by (most cited first, 30 of 30)

- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2105.10311/) (2021)
- [Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics](https://mlsanity.com/paper/2204.10216/) (2022)
- [Low-Resource Dialogue Summarization with Domain-Agnostic Multi-Source Pretraining](https://mlsanity.com/paper/2109.04080/) (2021)
- [CiteSum: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision](https://mlsanity.com/paper/2205.06207/) (2022)
- [UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization](https://mlsanity.com/paper/2211.09783/) (2022)
- [Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarization](https://mlsanity.com/paper/2204.04362/) (2022)
- [PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization](https://mlsanity.com/paper/2204.04413/) (2022)
- [A Character-Level Length-Control Algorithm for Non-Autoregressive Sentence Summarization](https://mlsanity.com/paper/2205.14522/) (2022)
- [Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization](https://mlsanity.com/paper/2205.14521/) (2022)
- [Towards Summary Candidates Fusion](https://mlsanity.com/paper/2210.08779/) (2022)
- [Toward Unified Controllable Text Generation via Regular Expression Instruction](https://mlsanity.com/paper/2309.10447/) (2023)
- [Information-Theoretic Distillation for Reference-less Summarization](https://mlsanity.com/paper/2403.13780/) (2024)
- [Unsupervised Multi-Granularity Summarization](https://mlsanity.com/paper/2201.12502/) (2022)
- [EASE: Extractive-Abstractive Summarization with Explanations](https://mlsanity.com/paper/2105.06982/) (2021)
- [Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation](https://mlsanity.com/paper/2107.13077/) (2021)
- [An Efficient Coarse-to-Fine Facet-Aware Unsupervised Summarization Framework based on Semantic Blocks](https://mlsanity.com/paper/2208.08253/) (2022)
- [Understanding Position Bias Effects on Fairness in Social Multi-Document Summarization](https://mlsanity.com/paper/2405.01790/) (2024)
- [Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuning](https://mlsanity.com/paper/2305.12077/) (2023)
- [Combining State-of-the-Art Models with Maximal Marginal Relevance for Few-Shot and Zero-Shot Multi-Document Summarization](https://mlsanity.com/paper/2211.10808/) (2022)
- [PROM: A Phrase-level Copying Mechanism with Pre-training for Abstractive Summarization](https://mlsanity.com/paper/2305.06647/) (2023)
- [i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data](https://mlsanity.com/paper/2305.12311/) (2023)
- [RTSUM: Relation Triple-based Interpretable Summarization with Multi-level Salience Visualization](https://mlsanity.com/paper/2310.13895/) (2023)
- [Unsupervised Extractive Summarization with Learnable Length Control Strategies](https://mlsanity.com/paper/2312.06901/) (2023)
- [Paired by the Teacher: Turning Unpaired Data into High-Fidelity Pairs for Low-Resource Text Generation](https://mlsanity.com/paper/2509.25144/) (2025)
- [Unsupervised Summarization Re-ranking](https://mlsanity.com/paper/2212.09593/) (2022)
- [Improving Sentence Similarity Estimation for Unsupervised Extractive Summarization](https://mlsanity.com/paper/2302.12490/) (2023)
- [Query-Guided Self-Supervised Summarization of Nursing Notes](https://mlsanity.com/paper/2407.04125/) (2024)
- [BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion](https://mlsanity.com/paper/2603.11415/) (2026)

## Abstract (arXiv)

Text summarization aims to extract essential information from a piece of text and transform the text into a concise version. Existing unsupervised abstractive summarization models leverage recurrent neural networks framework while the recently proposed transformer exhibits much more capability. Moreover, most of previous summarization models ignore abundant unlabeled corpora resources available for pretraining. In order to address these issues, we propose TED, a transformer-based unsupervised abstractive summarization system with pretraining on large-scale data. We first leverage the lead bias in news articles to pretrain the model on millions of unlabeled corpora. Next, we finetune TED on target domains through theme modeling and a denoising autoencoder to enhance the quality of generated summaries. Notably, TED outperforms all unsupervised abstractive baselines on NYT, CNN/DM and English Gigaword datasets with various document styles. Further analysis shows that the summaries generated by TED are highly abstractive, and each component in the objective function of TED is highly effective.