# Vector Quantized Wasserstein Auto-Encoder

arXiv: https://arxiv.org/abs/2302.05917 · HTML: https://mlsanity.com/paper/2302.05917/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 59 references that are in the map.
- First posted on arXiv 12 Feb 2023.

## Builds on (most cited first, 30 of 37)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Stochastic Optimization for Large-scale Optimal Transport](https://mlsanity.com/paper/1605.08527/) (2016)
- [MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation](https://mlsanity.com/paper/2209.09002/) (2022)
- [Video Pixel Networks](https://mlsanity.com/paper/1610.00527/) (2016)
- [Parallel Multiscale Autoregressive Density Estimation](https://mlsanity.com/paper/1703.03664/) (2017)
- [SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization](https://mlsanity.com/paper/2205.07547/) (2022)
- [The challenge of realistic music generation: modelling raw audio at scale](https://mlsanity.com/paper/1806.10474/) (2018)
- [Hierarchical Quantized Autoencoders](https://mlsanity.com/paper/2002.08111/) (2020)
- [Estimating or Propagating Gradients Through Stochastic Neurons](https://mlsanity.com/paper/1305.2982/) (2013)
- [Theory and Experiments on Vector Quantized Autoencoders](https://mlsanity.com/paper/1805.11063/) (2018)

## Built on by (most cited first, 22 of 22)

- [Addressing Representation Collapse in Vector Quantized Models with One Linear Layer](https://mlsanity.com/paper/2411.02038/) (2024)
- [Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%](https://mlsanity.com/paper/2406.11837/) (2024)
- [Online Clustered Codebook](https://mlsanity.com/paper/2307.15139/) (2023)
- [Transformed Distribution Matching for Missing Value Imputation](https://mlsanity.com/paper/2302.10363/) (2023)
- [Tuning Multi-mode Token-level Prompt Alignment across Modalities](https://mlsanity.com/paper/2309.13847/) (2023)
- [Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling](https://mlsanity.com/paper/2403.10071/) (2024)
- [Image Understanding Makes for A Good Tokenizer for Image Generation](https://mlsanity.com/paper/2411.04406/) (2024)
- [LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models](https://mlsanity.com/paper/2406.09008/) (2024)
- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [Optimal Transport for Structure Learning Under Missing Data](https://mlsanity.com/paper/2402.15255/) (2024)
- [Enhancing Vector Quantization with Distributional Matching: A Theoretical and Empirical Study](https://mlsanity.com/paper/2506.15078/) (2025)
- [Preserving Clusters in Prompt Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2506.11493/) (2025)
- [Rate-Adaptive Quantization: A Multi-Rate Codebook Adaptation for Vector Quantization-based Generative Models](https://mlsanity.com/paper/2405.14222/) (2024)
- [Discrete Distribution Networks](https://mlsanity.com/paper/2401.00036/) (2023)
- [Why Domain Generalization Fail? A View of Necessity and Sufficiency](https://mlsanity.com/paper/2502.10716/) (2025)
- [Parameter Estimation in DAGs from Incomplete Data via Optimal Transport](https://mlsanity.com/paper/2305.15927/) (2023)
- [Distributional Matching for Vector Quantization: A Unified Theoretical and Empirical Framework](https://mlsanity.com/paper/2607.15933/) (2026)
- [VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers](https://mlsanity.com/paper/2607.19575/) (2026)
- [Dual Codebook VQ: Enhanced Image Reconstruction with Reduced Codebook Size](https://mlsanity.com/paper/2503.10832/) (2025)
- [Task-Driven Discrete Representation Learning](https://mlsanity.com/paper/2506.11511/) (2025)
- [From Points to Spheres: A Geometric Reinterpretation of Variational Autoencoders](https://mlsanity.com/paper/2507.17255/) (2025)
- [ArcVQ-VAE: A Spherical Vector Quantization Framework with ArcCosine Additive Margin](https://mlsanity.com/paper/2605.13517/) (2026)

## Abstract (arXiv)

Learning deep discrete latent presentations offers a promise of better symbolic and summarized abstractions that are more useful to subsequent downstream tasks. Inspired by the seminal Vector Quantized Variational Auto-Encoder (VQ-VAE), most of work in learning deep discrete representations has mainly focused on improving the original VQ-VAE form and none of them has studied learning deep discrete representations from the generative viewpoint. In this work, we study learning deep discrete representations from the generative viewpoint. Specifically, we endow discrete distributions over sequences of codewords and learn a deterministic decoder that transports the distribution over the sequences of codewords to the data distribution via minimizing a WS distance between them. We develop further theories to connect it with the clustering viewpoint of WS distance, allowing us to have a better and more controllable clustering solution. Finally, we empirically evaluate our method on several well-known benchmarks, where it achieves better qualitative and quantitative performances than the other VQ-VAE variants in terms of the codebook utilization and image reconstruction/generation.