# Relaxed-Responsibility Hierarchical Discrete VAEs

arXiv: https://arxiv.org/abs/2007.07307 · HTML: https://mlsanity.com/paper/2007.07307/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 4 of 4)

- [Lossy Image Compression with Quantized Hierarchical VAEs](https://mlsanity.com/paper/2208.13056/) (2022)
- [Hierarchical Sketch Induction for Paraphrase Generation](https://mlsanity.com/paper/2203.03463/) (2022)
- [I Don't Need u: Identifiable Non-Linear ICA Without Side Information](https://mlsanity.com/paper/2106.05238/) (2021)
- [Diffusion bridges vector quantized Variational AutoEncoders](https://mlsanity.com/paper/2202.04895/) (2022)

## Abstract (arXiv)

Successfully training Variational Autoencoders (VAEs) with a hierarchy of discrete latent variables remains an area of active research. Vector-Quantised VAEs are a powerful approach to discrete VAEs, but naive hierarchical extensions can be unstable when training. Leveraging insights from classical methods of inference we introduce textitRelaxed-Responsibility Vector-Quantisation, a novel way to parameterise discrete latent variables, a refinement of relaxed Vector-Quantisation that gives better performance and more stable training. This enables a novel approach to hierarchical discrete variational autoencoders with numerous layers of latent variables (here up to 32) that we train end-to-end. Within hierarchical probabilistic deep generative models with discrete latent variables trained end-to-end, we achieve state-of-the-art bits-per-dim results for various standard datasets. % Unlike discrete VAEs with a single layer of latent variables, we can produce samples by ancestral sampling: it is not essential to train a second autoregressive generative model over the learnt latent representations to then sample from and then decode. % Moreover, that latter approach in these deep hierarchical models would require thousands of forward passes to generate a single sample. Further, we observe different layers of our model become associated with different aspects of the data.