# Language as a Latent Variable: Discrete Generative Models for Sentence Compression

arXiv: https://arxiv.org/abs/1609.07317 · HTML: https://mlsanity.com/paper/1609.07317/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2016.

## Built on by (most cited first, 28 of 28)

- [Re3: Generating Longer Stories With Recursive Reprompting and Revision](https://mlsanity.com/paper/2210.06774/) (2022)
- [Joint Prompt Optimization of Stacked LLMs using Variational Inference](https://mlsanity.com/paper/2306.12509/) (2023)
- [Why is constrained neural language generation particularly challenging?](https://mlsanity.com/paper/2206.05395/) (2022)
- [See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization](https://mlsanity.com/paper/2105.09601/) (2021)
- [Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning](https://mlsanity.com/paper/2205.08221/) (2022)
- [Automatic Text Summarization Methods: A Comprehensive Review](https://mlsanity.com/paper/2204.01849/) (2022)
- [Guiding Visual Question Generation](https://mlsanity.com/paper/2110.08226/) (2021)
- [Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation](https://mlsanity.com/paper/2210.13800/) (2022)
- [Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization](https://mlsanity.com/paper/2205.14521/) (2022)
- [Unsupervised Abstractive Summarization of Bengali Text Documents](https://mlsanity.com/paper/2102.04490/) (2021)
- [Cross-Modal Generative Augmentation for Visual Question Answering](https://mlsanity.com/paper/2105.04780/) (2021)
- [End-to-end Story Plot Generator](https://mlsanity.com/paper/2310.08796/) (2023)
- [Diffusion map particle systems for generative modeling](https://mlsanity.com/paper/2304.00200/) (2023)
- [Intra-agent speech permits zero-shot task acquisition](https://mlsanity.com/paper/2206.03139/) (2022)
- [EASE: Extractive-Abstractive Summarization with Explanations](https://mlsanity.com/paper/2105.06982/) (2021)
- [InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models](https://mlsanity.com/paper/2406.11097/) (2024)
- [May the Force Be with Your Copy Mechanism: Enhanced Supervised-Copy Method for Natural Language Generation](https://mlsanity.com/paper/2112.10360/) (2021)
- [CAISE: Conversational Agent for Image Search and Editing](https://mlsanity.com/paper/2202.11847/) (2022)
- [Learning with Latent Structures in Natural Language Processing: A Survey](https://mlsanity.com/paper/2201.00490/) (2022)
- [Reinforced Generative Adversarial Network for Abstractive Text Summarization](https://mlsanity.com/paper/2105.15176/) (2021)
- [Language as a Latent Sequence: deep latent variable models for semi-supervised paraphrase generation](https://mlsanity.com/paper/2301.02275/) (2023)
- [Topic-Aware Encoding for Extractive Summarization](https://mlsanity.com/paper/2112.09572/) (2021)
- [Adaptive Beam Search to Enhance On-device Abstractive Summarization](https://mlsanity.com/paper/2201.02739/) (2021)
- [Towards Computationally Verifiable Semantic Grounding for Language Models](https://mlsanity.com/paper/2211.09070/) (2022)
- [Scalable Learning of Latent Language Structure With Logical Offline Cycle Consistency](https://mlsanity.com/paper/2305.20018/) (2023)
- [Adversarial Conversational Shaping for Intelligent Agents](https://mlsanity.com/paper/2307.11785/) (2023)
- [From Lengthy to Lucid: A Systematic Literature Review on NLP Techniques for Taming Long Sentences](https://mlsanity.com/paper/2312.05172/) (2023)
- [Understanding Confabulation and Rethinking Reconstruction in Activation Explanations](https://mlsanity.com/paper/2609.33702/) (2026)

## Abstract (arXiv)

In this work we explore deep generative models of text in which the latent representation of a document is itself drawn from a discrete language model distribution. We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences. In this application the generative model first draws a latent summary sentence from a background language model, and then subsequently draws the observed sentence conditioned on this latent summary. In our empirical evaluation we show that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data. Further, we explore semi-supervised compression scenarios where we show that it is possible to achieve performance competitive with previously proposed supervised models while training on a fraction of the supervised data.