# Deleter: Leveraging BERT to Perform Unsupervised Successive Text Compression

arXiv: https://arxiv.org/abs/1909.03223 · HTML: https://mlsanity.com/paper/1909.03223/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2019.

## Built on by (most cited first, 5 of 5)

- [Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models](https://mlsanity.com/paper/2407.15504/) (2024)
- [Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning](https://mlsanity.com/paper/2205.08221/) (2022)
- [Less than One-shot: Named Entity Recognition via Extremely Weak Supervision](https://mlsanity.com/paper/2311.02861/) (2023)
- [InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models](https://mlsanity.com/paper/2406.11097/) (2024)
- [Contextualized Semantic Distance between Highly Overlapped Texts](https://mlsanity.com/paper/2110.01176/) (2021)

## Abstract (arXiv)

Text compression has diverse applications such as Summarization, Reading Comprehension and Text Editing. However, almost all existing approaches require either hand-crafted features, syntactic labels or parallel data. Even for one that achieves this task in an unsupervised setting, its architecture necessitates a task-specific autoencoder. Moreover, these models only generate one compressed sentence for each source input, so that adapting to different style requirements (e.g. length) for the final output usually implies retraining the model from scratch. In this work, we propose a fully unsupervised model, Deleter, that is able to discover an "optimal deletion path" for an arbitrary sentence, where each intermediate sequence along the path is a coherent subsequence of the previous one. This approach relies exclusively on a pretrained bidirectional language model (BERT) to score each candidate deletion based on the average Perplexity of the resulting sentence and performs progressive greedy lookahead search to select the best deletion for each step. We apply Deleter to the task of extractive Sentence Compression, and found that our model is competitive with state-of-the-art supervised models trained on 1.02 million in-domain examples with similar compression ratio. Qualitative analysis, as well as automatic and human evaluations both verify that our model produces high-quality compression.