# Sentence Bottleneck Autoencoders from Transformer Language Models

arXiv: https://arxiv.org/abs/2109.00055 · HTML: https://mlsanity.com/paper/2109.00055/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [From Variational to Deterministic Autoencoders](https://mlsanity.com/paper/1903.12436/) (2019)
- [Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation](https://mlsanity.com/paper/2006.10369/) (2020)
- [On the Sentence Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2011.05864/) (2020)
- [Content preserving text generation with attribute controls](https://mlsanity.com/paper/1811.01135/) (2018)
- [Controllable Unsupervised Text Attribute Transfer via Editing Entangled Latent Representation](https://mlsanity.com/paper/1905.12926/) (2019)
- [Plug and Play Autoencoders for Conditional Text Generation](https://mlsanity.com/paper/2010.02983/) (2020)

## Built on by (most cited first, 19 of 19)

- [Extracting Latent Steering Vectors from Pretrained Language Models](https://mlsanity.com/paper/2205.05124/) (2022)
- [Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity](https://mlsanity.com/paper/2502.13063/) (2025)
- [E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation](https://mlsanity.com/paper/2205.14912/) (2022)
- [Time Series Anomaly Detection using Diffusion-based Models](https://mlsanity.com/paper/2311.01452/) (2023)
- [Semformer: Transformer Language Models with Semantic Planning](https://mlsanity.com/paper/2409.11143/) (2024)
- [EncT5: A Framework for Fine-tuning T5 as Non-autoregressive Models](https://mlsanity.com/paper/2110.08426/) (2021)
- [Decoding a Neural Retriever's Latent Space for Query Suggestion](https://mlsanity.com/paper/2210.12084/) (2022)
- [Conversation Style Transfer using Few-Shot Learning](https://mlsanity.com/paper/2302.08362/) (2023)
- [SALSA: Semantically-Aware Latent Space Autoencoder](https://mlsanity.com/paper/2310.02744/) (2023)
- [vec2text with Round-Trip Translations](https://mlsanity.com/paper/2209.06792/) (2022)
- [Language Model Prompt Selection via Simulation Optimization](https://mlsanity.com/paper/2404.08164/) (2024)
- [Language Model Pre-Training with Sparse Latent Typing](https://mlsanity.com/paper/2210.12582/) (2022)
- [Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation](https://mlsanity.com/paper/2110.07002/) (2021)
- [Neuro-Symbolic Execution of Generic Source Code](https://mlsanity.com/paper/2304.00989/) (2023)
- [Towards Controllable Natural Language Inference through Lexical Inference Types](https://mlsanity.com/paper/2308.03581/) (2023)
- [Text Compression for Efficient Language Generation](https://mlsanity.com/paper/2503.11426/) (2025)
- [Data-efficient Performance Modeling via Pre-training](https://mlsanity.com/paper/2501.14438/) (2025)
- [Set-Theoretic Compositionality of Sentence Embeddings](https://mlsanity.com/paper/2502.20975/) (2025)
- [WeNLEX: Weakly Supervised Natural Language Explanations for Multilabel Chest X-ray Classification](https://mlsanity.com/paper/2603.18752/) (2026)

## Abstract (arXiv)

Representation learning for text via pretraining a language model on a large corpus has become a standard starting point for building NLP systems. This approach stands in contrast to autoencoders, also trained on raw text, but with the objective of learning to encode each input as a vector that allows full reconstruction. Autoencoders are attractive because of their latent space structure and generative properties. We therefore explore the construction of a sentence-level autoencoder from a pretrained, frozen transformer language model. We adapt the masked language modeling objective as a generative, denoising one, while only training a sentence bottleneck and a single-layer modified transformer decoder. We demonstrate that the sentence representations discovered by our model achieve better quality than previous methods that extract representations from pretrained transformers on text similarity tasks, style transfer (an example of controlled generation), and single-sentence classification tasks in the GLUE benchmark, while using fewer parameters than large pretrained models.