# Hierarchical Quantized Autoencoders

arXiv: https://arxiv.org/abs/2002.08111 · HTML: https://mlsanity.com/paper/2002.08111/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2020.

## Built on by (most cited first, 30 of 57)

- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanism](https://mlsanity.com/paper/2309.00796/) (2023)
- [Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks](https://mlsanity.com/paper/2305.08842/) (2023)
- [Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%](https://mlsanity.com/paper/2406.11837/) (2024)
- [Online Clustered Codebook](https://mlsanity.com/paper/2307.15139/) (2023)
- [Soft then Hard: Rethinking the Quantization in Neural Image Compression](https://mlsanity.com/paper/2104.05168/) (2021)
- [MoGenTS: Motion Generation based on Spatial-Temporal Joint Modeling](https://mlsanity.com/paper/2409.17686/) (2024)
- [DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion](https://mlsanity.com/paper/2309.01372/) (2023)
- [Lossy Image Compression with Quantized Hierarchical VAEs](https://mlsanity.com/paper/2208.13056/) (2022)
- [QARV: Quantization-Aware ResNet VAE for Lossy Image Compression](https://mlsanity.com/paper/2302.08899/) (2023)
- [T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text](https://mlsanity.com/paper/2406.07119/) (2024)
- [HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes](https://mlsanity.com/paper/2401.00365/) (2023)
- [Hierarchical Sketch Induction for Paraphrase Generation](https://mlsanity.com/paper/2203.03463/) (2022)
- [Vector Quantized Wasserstein Auto-Encoder](https://mlsanity.com/paper/2302.05917/) (2023)
- [EdVAE: Mitigating Codebook Collapse with Evidential Discrete Variational Autoencoders](https://mlsanity.com/paper/2310.05718/) (2023)
- [Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling](https://mlsanity.com/paper/2403.10071/) (2024)
- [Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learning](https://mlsanity.com/paper/2406.03234/) (2024)
- [StraIT: Non-autoregressive Generation with Stratified Image Transformer](https://mlsanity.com/paper/2303.00750/) (2023)
- [T2M-HiFiGPT: Generating High Quality Human Motion from Textual Descriptions with Residual Discrete Representations](https://mlsanity.com/paper/2312.10628/) (2023)
- [Hierarchical disentangled representation learning for singing voice conversion](https://mlsanity.com/paper/2101.06842/) (2021)
- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [GAITGen: Disentangled Motion-Pathology Impaired Gait Generative Model -- Bringing Motion Generation to the Clinical Domain](https://mlsanity.com/paper/2503.22397/) (2025)
- [Hierarchical Residual Learning Based Vector Quantized Variational Autoencoder for Image Reconstruction and Generation](https://mlsanity.com/paper/2208.04554/) (2022)
- [MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild](https://mlsanity.com/paper/2412.13393/) (2024)
- [Enhancing Vector Quantization with Distributional Matching: A Theoretical and Empirical Study](https://mlsanity.com/paper/2506.15078/) (2025)
- [MoSa: Motion Generation with Scalable Autoregressive Modeling](https://mlsanity.com/paper/2511.01200/) (2025)
- [Semantic Image Synthesis with Semantically Coupled VQ-Model](https://mlsanity.com/paper/2209.02536/) (2022)
- [Tokenization with Factorized Subword Encoding](https://mlsanity.com/paper/2306.07764/) (2023)
- [LL-VQ-VAE: Learnable Lattice Vector-Quantization For Efficient Representations](https://mlsanity.com/paper/2310.09382/) (2023)

## Abstract (arXiv)

Despite progress in training neural networks for lossy image compression, current approaches fail to maintain both perceptual quality and abstract features at very low bitrates. Encouraged by recent success in learning discrete representations with Vector Quantized Variational Autoencoders (VQ-VAEs), we motivate the use of a hierarchy of VQ-VAEs to attain high factors of compression. We show that the combination of stochastic quantization and hierarchical latent structure aids likelihood-based image compression. This leads us to introduce a novel objective for training hierarchical VQ-VAEs. Our resulting scheme produces a Markovian series of latent variables that reconstruct images of high-perceptual quality which retain semantically meaningful features. We provide qualitative and quantitative evaluations on the CelebA and MNIST datasets.