# Finetuning Pretrained Transformers into Variational Autoencoders

arXiv: https://arxiv.org/abs/2108.02446 · HTML: https://mlsanity.com/paper/2108.02446/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing](https://mlsanity.com/paper/1903.10145/) (2019)
- [Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space](https://mlsanity.com/paper/2004.04092/) (2020)
- [Semi-Amortized Variational Autoencoders](https://mlsanity.com/paper/1802.02550/) (2018)
- [Improved Variational Autoencoders for Text Modeling using Dilated Convolutions](https://mlsanity.com/paper/1702.08139/) (2017)
- [A Probabilistic Formulation of Unsupervised Text Style Transfer](https://mlsanity.com/paper/2002.03912/) (2020)
- [Transformer-based Conditional Variational Autoencoder for Controllable Story Generation](https://mlsanity.com/paper/2101.00828/) (2021)
- [A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text](https://mlsanity.com/paper/1909.00868/) (2019)
- [Re-balancing Variational Autoencoder Loss for Molecule Sequence Generation](https://mlsanity.com/paper/1910.00698/) (2019)
- [On Posterior Collapse and Encoder Feature Dispersion in Sequence VAEs](https://mlsanity.com/paper/1911.03976/) (2019)

## Built on by (most cited first, 14 of 14)

- [Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference](https://mlsanity.com/paper/2403.04082/) (2024)
- [Neuro-Symbolic Embedding for Short and Effective Feature Selection via Autoregressive Generation](https://mlsanity.com/paper/2404.17157/) (2024)
- [AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modeling](https://mlsanity.com/paper/2205.05862/) (2022)
- [Decoding a Neural Retriever's Latent Space for Query Suggestion](https://mlsanity.com/paper/2210.12084/) (2022)
- [PCAE: A Framework of Plug-in Conditional Auto-Encoder for Controllable Text Generation](https://mlsanity.com/paper/2210.03496/) (2022)
- [Latent Diffusion Models for Controllable RNA Sequence Generation](https://mlsanity.com/paper/2409.09828/) (2024)
- [Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment](https://mlsanity.com/paper/2412.20834/) (2024)
- [A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck](https://mlsanity.com/paper/2207.13529/) (2022)
- [vec2text with Round-Trip Translations](https://mlsanity.com/paper/2209.06792/) (2022)
- [Nonparametric Variational Regularisation of Pretrained Transformers](https://mlsanity.com/paper/2312.00662/) (2023)
- [Transformers as Neural Augmentors: Class Conditional Sentence Generation via Variational Bayes](https://mlsanity.com/paper/2205.09391/) (2022)
- [An Overview on Controllable Text Generation via Variational Auto-Encoders](https://mlsanity.com/paper/2211.07954/) (2022)
- [Harnessing Attention Mechanisms: Efficient Sequence Reduction using Attention-based Autoencoders](https://mlsanity.com/paper/2310.14837/) (2023)
- [Uncovering Latent Reasoning Strategies in Language Models](https://mlsanity.com/paper/2607.17674/) (2026)

## Abstract (arXiv)

Text variational autoencoders (VAEs) are notorious for posterior collapse, a phenomenon where the model's decoder learns to ignore signals from the encoder. Because posterior collapse is known to be exacerbated by expressive decoders, Transformers have seen limited adoption as components of text VAEs. Existing studies that incorporate Transformers into text VAEs (Li et al., 2020; Fang et al., 2021) mitigate posterior collapse using massive pretraining, a technique unavailable to most of the research community without extensive computing resources. We present a simple two-phase training scheme to convert a sequence-to-sequence Transformer into a VAE with just finetuning. The resulting language model is competitive with massively pretrained Transformer-based VAEs in some internal metrics while falling short on others. To facilitate training we comprehensively explore the impact of common posterior collapse alleviation techniques in the literature. We release our code for reproducability.