# BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principle

arXiv: https://arxiv.org/abs/1909.07405 · HTML: https://mlsanity.com/paper/1909.07405/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2019.

## Built on by (most cited first, 30 of 39)

- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [Variational Information Bottleneck for Effective Low-Resource Fine-Tuning](https://mlsanity.com/paper/2106.05469/) (2021)
- [Rethinking External Slow-Thinking: From Snowball Errors to Probability of Correct Reasoning](https://mlsanity.com/paper/2501.15602/) (2025)
- [Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens](https://mlsanity.com/paper/2505.18237/) (2025)
- [TimeX++: Learning Time-Series Explanations with Information Bottleneck](https://mlsanity.com/paper/2405.09308/) (2024)
- [Reason first, then respond: Modular Generation for Knowledge-infused Dialogue](https://mlsanity.com/paper/2111.05204/) (2021)
- [InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding](https://mlsanity.com/paper/2306.04933/) (2023)
- [Protecting Your LLMs with Information Bottleneck](https://mlsanity.com/paper/2404.13968/) (2024)
- [The Summary Loop: Learning to Write Abstractive Summaries Without Examples](https://mlsanity.com/paper/2105.05361/) (2021)
- [Unsupervised Extractive Summarization using Pointwise Mutual Information](https://mlsanity.com/paper/2102.06272/) (2021)
- [Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning](https://mlsanity.com/paper/2205.08221/) (2022)
- [Improving the Adversarial Robustness of NLP Models by Information Bottleneck](https://mlsanity.com/paper/2206.05511/) (2022)
- [An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.01549/) (2024)
- [Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective](https://mlsanity.com/paper/2410.01720/) (2024)
- [Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation](https://mlsanity.com/paper/2210.13800/) (2022)
- [Leveraging Information Bottleneck for Scientific Document Summarization](https://mlsanity.com/paper/2110.01280/) (2021)
- [A Character-Level Length-Control Algorithm for Non-Autoregressive Sentence Summarization](https://mlsanity.com/paper/2205.14522/) (2022)
- [Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization](https://mlsanity.com/paper/2205.14521/) (2022)
- [A Multi-Format Transfer Learning Model for Event Argument Extraction via Variational Information Bottleneck](https://mlsanity.com/paper/2208.13017/) (2022)
- [Dual Latent Memory for Visual Multi-agent System](https://mlsanity.com/paper/2602.00471/) (2026)
- [CO2Sum:Contrastive Learning for Factual-Consistent Abstractive Summarization](https://mlsanity.com/paper/2112.01147/) (2021)
- [TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Models](https://mlsanity.com/paper/2510.01274/) (2025)
- [NavigScene: Bridging Local Perception and Global Navigation for Beyond-Visual-Range Autonomous Driving](https://mlsanity.com/paper/2507.05227/) (2025)
- [LawSum: A weakly supervised approach for Indian Legal Document Summarization](https://mlsanity.com/paper/2110.01188/) (2021)
- [BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principle](https://mlsanity.com/paper/2502.18331/) (2025)
- [Generating Multiple-Length Summaries via Reinforcement Learning for Unsupervised Sentence Summarization](https://mlsanity.com/paper/2212.10843/) (2022)
- [EASE: Extractive-Abstractive Summarization with Explanations](https://mlsanity.com/paper/2105.06982/) (2021)
- [Explanation Regeneration via Information Bottleneck](https://mlsanity.com/paper/2212.09603/) (2022)
- [Span-based Named Entity Recognition by Generating and Compressing Information](https://mlsanity.com/paper/2302.05392/) (2023)
- [VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR](https://mlsanity.com/paper/2505.17423/) (2025)

## Abstract (arXiv)

The principle of the Information Bottleneck (Tishby et al. 1999) is to produce a summary of information X optimized to predict some other relevant information Y. In this paper, we propose a novel approach to unsupervised sentence summarization by mapping the Information Bottleneck principle to a conditional language modelling objective: given a sentence, our approach seeks a compressed sentence that can best predict the next sentence. Our iterative algorithm under the Information Bottleneck objective searches gradually shorter subsequences of the given sentence while maximizing the probability of the next sentence conditioned on the summary. Using only pretrained language models with no direct supervision, our approach can efficiently perform extractive sentence summarization over a large corpus. Building on our unsupervised extractive summarization (BottleSumEx), we then present a new approach to self-supervised abstractive summarization (BottleSumSelf), where a transformer-based language model is trained on the output summaries of our unsupervised method. Empirical results demonstrate that our extractive method outperforms other unsupervised models on multiple automatic metrics. In addition, we find that our self-supervised abstractive model outperforms unsupervised baselines (including our own) by human evaluation along multiple attributes.