mlsanityData to 14 Oct 2026

Finite Scalar Quantization: VQ-VAE Made Simple

arXiv 2309.15505 ↗First posted 27 Sep 2023cs.CV cs.LGcited by 440

Not on the map. Why?

Cited by 59 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose to replace vector quantization (VQ) in the latent representation of VQ-VAEs with a simple scheme termed finite scalar quantization (FSQ), where we project the VAE representation down to a few dimensions (typically less than 10). Each dimension is quantized to a small set of fixed values, leading to an (implicit) codebook given by the product of these sets. By appropriately choosing the number of dimensions and values each dimension can take, we obtain the same codebook size as in VQ. On top of such discrete representations, we can train the same models that have been trained on VQ-VAE representations. For example, autoregressive and masked transformer models for image generation, multimodal generation, and dense prediction computer vision tasks. Concretely, we employ FSQ with MaskGIT for image generation, and with UViM for depth estimation, colorization, and panoptic segmentation. Despite the much simpler design of FSQ, we obtain competitive performance in all these tasks. We emphasize that FSQ does not suffer from codebook collapse and does not need the complex machinery employed in VQ (commitment losses, codebook reseeding, code splitting, entropy penalties, etc.) to learn expressive discrete representations.

Key facts. Cited by 440 papers in the map (data through 14 Oct 2026). Builds on 45 of its 51 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

45 of 51 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  11. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  12. 2016End-to-end Optimized Image Compressioncited by 696
33 more
  1. 2018Joint Autoregressive and Hierarchical Priors for Learned Image Compressioncited by 664
  2. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  3. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  4. 2020Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modulescited by 590
  5. 2020Jukebox: A Generative Model for Musiccited by 542
  6. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  7. 2018Assessing Generative Models via Precision and Recallcited by 406
  8. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  9. 2020High-Fidelity Generative Image Compressioncited by 338
  10. 2018DVC: An End-to-end Deep Video Compression Frameworkcited by 332
  11. 2017Lossy Image Compression with Compressive Autoencoderscited by 294
  12. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  13. 2018Generative Adversarial Networks for Extreme Learned Image Compressioncited by 219
  14. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  15. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  16. 2018Conditional Probability Models for Deep Image Compressioncited by 133
  17. 2018High Quality Monocular Depth Estimation via Transfer Learningcited by 133
  18. 2021Colorization Transformercited by 89
  19. 2023Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networkscited by 82
  20. 2022UViM: A Unified Modeling Approach for Vision with Learned Guiding Codescited by 78
  21. 2022SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantizationcited by 68
  22. 2018Deep Generative Models for Distribution-Preserving Lossy Compressioncited by 59
  23. 2020Hierarchical Quantized Autoencoderscited by 57
  24. 2018Theory and Experiments on Vector Quantized Autoencoderscited by 55
  25. 2020Robust Training of Vector Quantized Bottleneck Modelscited by 54
  26. 2023Disentanglement via Latent Quantizationcited by 40
  27. 2019Differentiable Product Quantization for End-to-End Embedding Compressioncited by 36
  28. 2022Image Compression with Product Quantized Masked Image Modelingcited by 33
  29. 2023Not All Image Regions Matter: Masked Vector Quantization for Autoregressive Image Generationcited by 28
  30. 2021Variable-rate discrete representation learningcited by 24
  31. 2023MAGVLT: Masked Generative Vision-and-Language Transformercited by 14
  32. 2023M2T: Masking Transformers Twice for Faster Decodingcited by 14
  33. 2018Piano Geniecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 85; 2025: 174; 2026: 180.

440 papers.

Sort:

By area:

  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  2. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  3. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  4. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  5. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  6. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  7. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  8. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  9. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  10. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  11. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  12. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 132
38 more
  1. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  2. 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 112
  3. 2025TerraMind: Large-Scale Generative Multimodality for Earth Observationcited by 104
  4. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  5. 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 101
  6. 2024MaskBit: Embedding-free Image Generation via Bit Tokenscited by 98
  7. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  8. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  9. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 82
  10. 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
  11. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 80
  12. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
  13. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 64
  14. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 56
  15. 2024ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Modelcited by 53
  16. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  17. 2024Restructuring Vector Quantization with the Rotation Trickcited by 51
  18. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  19. 2024ElasticTok: Adaptive Tokenization for Image and Videocited by 46
  20. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Textcited by 46
  21. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 45
  22. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 44
  23. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 43
  24. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  25. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 40
  26. 2024Autoregressive Models in Vision: A Surveycited by 39
  27. 2025M-Prometheus: A Suite of Open Multilingual LLM Judgescited by 38
  28. 2025Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenizationcited by 36
  29. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 35
  30. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  31. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 34
  32. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  33. 2025Halton Scheduler For Masked Generative Image Transformercited by 32
  34. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  35. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agentscited by 30
  36. 2025AToken: A Unified Tokenizer for Visioncited by 30
  37. 2025Bridging Continuous and Discrete Tokens for Autoregressive Visual Generationcited by 29
  38. 2025IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech Systemcited by 28

50 most cited of 440 All 440 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Scalable Image Tokenization with Index Backpropagation Quantization
  2. 2023Soft Convex Quantization: Revisiting Vector Quantization with Convex Optimization
  3. 2024HyperVQ: MLR-based Vector Quantization in Hyperbolic Space
  4. 2026MGVQ: Synergizing Multi-dimensional Sensitivity-Aware and Gradient-Hessian Fusion for Vector Quantization
  5. 2025VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
  6. 2023HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes
  7. 2023Vector Quantized Wasserstein Auto-Encoder
  8. 2026VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers