mlsanityData to 7 Oct 2026

SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization

arXiv 2205.07547 ↗First posted 16 May 2022cs.LG cs.CVcited by 68

Not on the map. Why?

Abstract (arXiv)

One noted issue of vector-quantized variational autoencoder (VQ-VAE) is that the learned discrete representation uses only a fraction of the full capacity of the codebook, also known as codebook collapse. We hypothesize that the training scheme of VQ-VAE, which involves some carefully designed heuristics, underlies this issue. In this paper, we propose a new training scheme that extends the standard VAE via novel stochastic dequantization and quantization, called stochastically quantized variational autoencoder (SQ-VAE). In SQ-VAE, we observe a trend that the quantization is stochastic at the initial stage of the training but gradually converges toward a deterministic quantization, which we call self-annealing. Our experiments show that SQ-VAE improves codebook utilization without using common heuristics. Furthermore, we empirically show that SQ-VAE is superior to VAE and VQ-VAE in vision- and speech-related tasks.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 16 May 2022.

Builds on

34 of 50 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
22 more
  1. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  2. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  3. 2016Pixel Recurrent Neural Networkscited by 814
  4. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  5. 2020Jukebox: A Generative Model for Musiccited by 535
  6. 2017Wasserstein Auto-Encoderscited by 371
  7. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  8. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  9. 2017Lossy Image Compression with Compressive Autoencoderscited by 289
  10. 2018Hyperspherical Variational Auto-Encoderscited by 188
  11. 2015Variable Rate Image Compression with Recurrent Neural Networkscited by 168
  12. 2017Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representationscited by 149
  13. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137
  14. 2019From Variational to Deterministic Autoencoderscited by 131
  15. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 103
  16. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  17. 2018Theory and Experiments on Vector Quantized Autoencoderscited by 55
  18. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  19. 2019The Zero Resource Speech Challenge 2019: TTS without Tcited by 38
  20. 2020Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challengecited by 21
  21. 2019Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networkscited by 20
  22. 2020Deterministic Decoding for Discrete Data in Variational Autoencoderscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 17; 2025: 20; 2026: 16.

68 papers.

Sort:

By area:

  1. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  2. 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 110
  3. 2023Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networkscited by 80
  4. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 77
  5. 2023Online Clustered Codebookcited by 55
  6. 2024Restructuring Vector Quantization with the Rotation Trickcited by 49
  7. 2023HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayescited by 27
  8. 2023Vector Quantized Wasserstein Auto-Encodercited by 22
  9. 2023Grounded Object Centric Learningcited by 17
  10. 2023EdVAE: Mitigating Codebook Collapse with Evidential Discrete Variational Autoencoderscited by 16
  11. 2024Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modelingcited by 16
  12. 2023SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layercited by 14
38 more
  1. 2024Image Understanding Makes for A Good Tokenizer for Image Generationcited by 14
  2. 2025Cube: A Roblox View of 3D Intelligencecited by 14
  3. 2024HyperVQ: MLR-based Vector Quantization in Hyperbolic Spacecited by 13
  4. 2026Continuous Latent Diffusion Language Modelcited by 12
  5. 2023StraIT: Non-autoregressive Generation with Stratified Image Transformercited by 10
  6. 2023Soft Convex Quantization: Revisiting Vector Quantization with Convex Optimizationcited by 10
  7. 2024Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learningcited by 10
  8. 2025MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantizationcited by 9
  9. 2024Unleash LLMs Potential for Sequential Recommendation by Coordinating Dual Dynamic Index Mechanismcited by 8
  10. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  11. 2023Learning Hierarchical Features with Joint Latent Space Energy-Based Priorcited by 6
  12. 2024Music Foundation Model as Generic Booster for Music Downstream Taskscited by 6
  13. 2025Improving 3D Gaussian Splatting Compression by Scene-Adaptive Lattice Vector Quantizationcited by 6
  14. 2025Enhancing Vector Quantization with Distributional Matching: A Theoretical and Empirical Studycited by 5
  15. 2025Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generationcited by 5
  16. 2025AI Mother Tongue: Self-Emergent Communication in MARL via Endogenous Symbol Systemscited by 5
  17. 2025DiVeQ: Differentiable Vector Quantization Using the Reparameterization Trickcited by 5
  18. 2025VAEVQ: Enhancing Discrete Visual Tokenization through Variational Modelingcited by 5
  19. 2023A Unified Framework for Generative Data Augmentation: A Comprehensive Surveycited by 4
  20. 2023LL-VQ-VAE: Learnable Lattice Vector-Quantization For Efficient Representationscited by 4
  21. 2024Balance of Number of Embedding and their Dimensions in Vector Quantizationcited by 4
  22. 2025Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Textcited by 4
  23. 2025Moment Quantization for Video Temporal Groundingcited by 4
  24. 2026Democratizing planetary-scale analysis: An ultra-lightweight Earth embedding database for accurate and flexible global land monitoringcited by 4
  25. 2024Weakly Augmented Variational Autoencoder in Time Series Anomaly Detectioncited by 2
  26. 2024Rate-Adaptive Quantization: A Multi-Rate Codebook Adaptation for Vector Quantization-based Generative Modelscited by 2
  27. 2024Trajectory Forecasting through Low-Rank Adaptation of Discrete Latent Codescited by 2
  28. 2024Gaussian Mixture Vector Quantization with Aggregated Categorical Posteriorcited by 2
  29. 2025Discrete JEPA: Learning Discrete Token Representations without Reconstructioncited by 2
  30. 2025Graph is a Natural Regularization: Revisiting Vector Quantization for Graph Representation Learningcited by 2
  31. 2023Parameter Estimation in DAGs from Incomplete Data via Optimal Transportcited by 1
  32. 2023Enhancing Semantic Communication with Deep Generative Models -- An ICASSP Special Session Overviewcited by 1
  33. 2024Efficient Video Face Enhancement with Enhanced Spatial-Temporal Consistencycited by 1
  34. 2025Cross-Layer Discrete Concept Discovery for Interpreting Language Modelscited by 1
  35. 2025Training-Free Vector Quantization via Gaussian VAEscited by 1
  36. 2026VP-VAE: Rethinking Vector Quantization via Adaptive Vector Perturbationcited by 1
  37. 2026Distributional Matching for Vector Quantization: A Unified Theoretical and Empirical Frameworkcited by 1
  38. 2024Using High-Level Patterns to Estimate How Humans Predict a Robot will Behavecited by 0

50 most cited of 68 All 68 (JSON)