# A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck

arXiv: https://arxiv.org/abs/2207.13529 · HTML: https://mlsanity.com/paper/2207.13529/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 27 Jul 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Transformer-based Conditional Variational Autoencoder for Controllable Story Generation](https://mlsanity.com/paper/2101.00828/) (2021)
- [Variational Transformers for Diverse Response Generation](https://mlsanity.com/paper/2003.12738/) (2020)
- [Finetuning Pretrained Transformers into Variational Autoencoders](https://mlsanity.com/paper/2108.02446/) (2021)
- [The Unstoppable Rise of Computational Linguistics in Deep Learning](https://mlsanity.com/paper/2005.06420/) (2020)
- [Stochastic gradient variational Bayes for gamma approximating distributions](https://mlsanity.com/paper/1509.01631/) (2015)
- [Eval all, trust a few, do wrong to none: Comparing sentence generation models](https://mlsanity.com/paper/1804.07972/) (2018)
- [Adversarially Regularized Autoencoders](https://mlsanity.com/paper/1706.04223/) (2017)
- [Educating Text Autoencoders: Latent Representation Guidance via Denoising](https://mlsanity.com/paper/1905.12777/) (2019)

## Built on by (most cited first, 3 of 3)

- [Massive Memorization with Hundreds of Trillions of Parameters for Sequential Transducer Generative Recommenders](https://mlsanity.com/paper/2510.22049/) (2025)
- [VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruning](https://mlsanity.com/paper/2406.05276/) (2024)
- [Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modeling](https://mlsanity.com/paper/2609.30288/) (2026)

## Abstract (arXiv)

We propose a VAE for Transformers by developing a variational information bottleneck regulariser for Transformer embeddings. We formalise the embedding space of Transformer encoders as mixture probability distributions, and use Bayesian nonparametrics to derive a nonparametric variational information bottleneck (NVIB) for such attention-based embeddings. The variable number of mixture components supported by nonparametric methods captures the variable number of vectors supported by attention, and the exchangeability of our nonparametric distributions captures the permutation invariance of attention. This allows NVIB to regularise the number of vectors accessible with attention, as well as the amount of information in individual vectors. By regularising the cross-attention of a Transformer encoder-decoder with NVIB, we propose a nonparametric variational autoencoder (NVAE). Initial experiments on training a NVAE on natural language text show that the induced embedding space has the desired properties of a VAE for Transformers.