mlsanityData to 14 Oct 2026

Online Clustered Codebook

arXiv 2307.15139 ↗First posted 27 Jul 2023cs.CVcited by 55

Not on the map. Why?

Abstract (arXiv)

Vector Quantisation (VQ) is experiencing a comeback in machine learning, where it is increasingly used in representation learning. However, optimizing the codevectors in existing VQ-VAE is not entirely trivial. A problem is codebook collapse, where only a small subset of codevectors receive gradients useful for their optimisation, whereas a majority of them simply ``dies off'' and is never updated or used. This limits the effectiveness of VQ for learning larger codebooks in complex computer vision tasks that require high-capacity representations. In this paper, we present a simple alternative method for online codebook learning, Clustering VQ-VAE (CVQ-VAE). Our approach selects encoded features as anchors to update the ``dead'' codevectors, while optimising the codebooks which are alive via the original loss. This strategy brings unused codevectors closer in distribution to the encoded features, increasing the likelihood of being chosen and optimized. We extensively validate the generalization capability of our quantiser on various datasets, tasks (e.g. reconstruction and generation), and architectures (e.g. VQ-VAE, VQGAN, LDM). Our CVQ-VAE can be easily integrated into the existing models with just a few lines of code.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 27 Jul 2023.

Builds on

37 of 47 references.

References · 37

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  11. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  12. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
25 more
  1. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  2. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  3. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  4. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  5. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  6. 2020Jukebox: A Generative Model for Musiccited by 542
  7. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  8. 2021Projected GANs Converge Fastercited by 208
  9. 2022AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generationcited by 208
  10. 2017Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representationscited by 152
  11. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125
  12. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  13. 2020Latent Video Transformercited by 106
  14. 2022SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantizationcited by 68
  15. 2020Hierarchical Quantized Autoencoderscited by 57
  16. 2021Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generationcited by 40
  17. 2022Unified Discrete Diffusion for Simultaneous Vision-Language Generationcited by 39
  18. 2021Cross-Modal Discrete Representation Learningcited by 35
  19. 2021Discrete Representations Strengthen Vision Transformer Robustnesscited by 35
  20. 2023VQ3D: Learning a 3D-Aware Generative Model on ImageNetcited by 32
  21. 2022Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesiscited by 31
  22. 2023Vector Quantized Wasserstein Auto-Encodercited by 22
  23. 2022UNIMO-2: End-to-End Unified Vision-Language Grounded Learningcited by 19
  24. 2023Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generationcited by 14
  25. 2022High-Quality Pluralistic Image Completion via Code Shared VQGANcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 23; 2026: 18.

55 papers.

Sort:

By area:

  1. 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 112
  2. 2024Loong: Generating Minute-level Long Videos with Autoregressive Language Modelscited by 99
  3. 2024MaskBit: Embedding-free Image Generation via Bit Tokenscited by 98
  4. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68
  5. 2024Restructuring Vector Quantization with the Rotation Trickcited by 51
  6. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  7. 2025CodeBrain: Bridging Decoupled Tokenizer and Multi-Scale Architecture for EEG Foundation Modelcited by 28
  8. 2024Stratified Avatar Generation from Sparse Observationscited by 21
  9. 2025Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Modelcited by 20
  10. 2025Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Spacecited by 16
  11. 2024Image Understanding Makes for A Good Tokenizer for Image Generationcited by 15
  12. 2025Scalable Training for Vector-Quantized Networks with 100% Codebook Utilizationcited by 10
38 more
  1. 2026Autoregressive Image Generation with Masked Bit Modelingcited by 9
  2. 2024Anytime Continual Learning for Open Vocabulary Classificationcited by 8
  3. 2025Qinco2: Vector Compression and Search with Improved Implicit Neural Codebookscited by 8
  4. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  5. 2025Enhancing Vector Quantization with Distributional Matching: A Theoretical and Empirical Studycited by 5
  6. 2025DiVeQ: Differentiable Vector Quantization Using the Reparameterization Trickcited by 5
  7. 2024Preventing Model Collapse in Gaussian Process Latent Variable Modelscited by 4
  8. 2024Balance of Number of Embedding and their Dimensions in Vector Quantizationcited by 4
  9. 2025Conan: A Chunkwise Online Network for Zero-Shot Adaptive Voice Conversioncited by 4
  10. 2024Rate-Adaptive Quantization: A Multi-Rate Codebook Adaptation for Vector Quantization-based Generative Modelscited by 3
  11. 2025Assessing the Completeness of Traffic Scenario Categories for Automated Highway Driving Functions via Cluster-based Analysiscited by 3
  12. 2025OneVAE: Joint Discrete and Continuous Optimization Helps Discrete Video VAE Train Bettercited by 3
  13. 2024Di2Pose: Discrete Diffusion Model for Occluded 3D Human Pose Estimationcited by 2
  14. 2024CoBooM: Codebook Guided Bootstrapping for Medical Image Representation Learningcited by 2
  15. 2025MARRS: Masked Autoregressive Unit-based Reaction Synthesiscited by 2
  16. 2025Hierarchical Characterization of Brain Dynamics via State Space-based Vector Quantizationcited by 2
  17. 2025Graph is a Natural Regularization: Revisiting Vector Quantization for Graph Representation Learningcited by 2
  18. 2025Exploiting Discriminative Codebook Prior for Autoregressive Image Generationcited by 2
  19. 2025Multi-View Oriented GPLVM: Expressiveness and Efficiencycited by 1
  20. 2026CtD: Composition through Decomposition in Emergent Communicationcited by 1
  21. 2026VP-VAE: Rethinking Vector Quantization via Adaptive Vector Perturbationcited by 1
  22. 2026Semantic-Aware Prefix Learning for Token-Efficient Image Generationcited by 1
  23. 2026Distributional Matching for Vector Quantization: A Unified Theoretical and Empirical Frameworkcited by 1
  24. 2026VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizerscited by 1
  25. 2026P2E-VQ: ECG-linked representation augmentation for PPG via discrete patch retrievalcited by 1
  26. 2026Flow-Matching-Based Protein Structure Tokenizer Made Efficient and Easycited by 1
  27. 2024Scalable Random Feature Latent Variable Modelscited by 0
  28. 2025Dual Codebook VQ: Enhanced Image Reconstruction with Reduced Codebook Sizecited by 0
  29. 2025D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokenscited by 0
  30. 2025POET: Prompt Offset Tuning for Continual Human Action Adaptationcited by 0
  31. 2025UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generationcited by 0
  32. 2025Generative AI-Driven High-Fidelity Human Motion Simulationcited by 0
  33. 2025MASC: Boosting Autoregressive Image Generation with a Manifold-Aligned Semantic Clusteringcited by 0
  34. 2026LooC: Effective Low-Dimensional Codebook for Compositional Vector Quantizationcited by 0
  35. 2026HanoiWorld : A Joint Embedding Predictive Architecture BasedWorld Model for Autonomous Vehicle Controllercited by 0
  36. 2026Generalized Radius and Integrated Codebook Transforms for Differentiable Vector Quantizationcited by 0
  37. 2026Efficient Anti-exploration via VQVAE and Fuzzy Clustering in Offline Reinforcement Learningcited by 0
  38. 2026Uncertainty-Aware Diffusion Model for Multimodal Highway Trajectory Prediction via DDIM Samplingcited by 0

50 most cited of 55 All 55 (JSON)