mlsanityData to 14 Oct 2026

Scalable Image Tokenization with Index Backpropagation Quantization

arXiv 2412.02692 ↗First posted 3 Dec 2024cs.CV cs.AIcited by 44

Not on the map. Why?

Abstract (arXiv)

Existing vector quantization (VQ) methods struggle with scalability, largely attributed to the instability of the codebook that undergoes partial updates during training. The codebook is prone to collapse as utilization decreases, due to the progressively widening distribution gap between non-activated codes and visual features. To solve the problem, we propose Index Backpropagation Quantization (IBQ), a new VQ method for the joint optimization of all codebook embeddings and the visual encoder. Applying a straight-through estimator on the one-hot categorical distribution between the encoded feature and codebook, all codes are differentiable and maintain a consistent latent space with the visual encoder. IBQ enables scalable training of visual tokenizers and, for the first time, achieves a large-scale codebook (218) with high dimension (256) and high utilization. Experiments on the standard ImageNet benchmark demonstrate the scalability and superiority of IBQ, achieving competitive results on reconstruction and the application of autoregressive visual generation. The code and models are available at https://github.com/TencentARC/SEED-Voken.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 42 of its 53 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

42 of 53 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2022Scalable Diffusion Models with Transformerscited by 6,054
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
30 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  3. 2016Improved Techniques for Training GANscited by 2,970
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  9. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  10. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  11. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  12. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  13. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  14. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  15. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  16. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  17. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  18. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  19. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  20. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  21. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  22. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  23. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  24. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  25. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
  26. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 155
  27. 2021Regularizing Generative Adversarial Networks under Limited Datacited by 107
  28. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  29. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68
  30. 2023Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantizationcited by 60

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 30.

44 papers.

Sort:

By area:

  1. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 132
  2. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 70
  3. 2025Uniform Discrete Diffusion with Metric Path for Video Generationcited by 12
  4. 2025MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptationcited by 10
  5. 2025VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstructioncited by 10
  6. 2025D-AR: Diffusion via Autoregressive Modelscited by 9
  7. 2025WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstructioncited by 9
  8. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  9. 2024Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generationcited by 7
  10. 2025DINO-Tok: Adapting DINO for Visual Tokenizerscited by 6
  11. 2025Spherical Leech Quantization for Visual Tokenization and Generationcited by 6
  12. 2026ResTok: Learning Hierarchical Residuals in 1D Visual Tokenizers for Autoregressive Image Generationcited by 5
32 more
  1. 2026End-to-End Autoregressive Image Generation with 1D Semantic Tokenizercited by 5
  2. 2026Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unificationcited by 4
  3. 2025ReDDiT: Rehashing Noise for Discrete Visual Generationcited by 3
  4. 2026Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generationcited by 3
  5. 2025Visual Generation Tuningcited by 2
  6. 2026EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generationcited by 2
  7. 2026MacTok: Robust Continuous Tokenization for Image Generationcited by 2
  8. 2025Training-Free Vector Quantization via Gaussian VAEscited by 1
  9. 2026UniGRec: Unified Generative Recommendation with Soft Identifiers for End-to-End Optimizationcited by 1
  10. 2026SNCE: Geometry-Aware Supervision for Scalable Discrete Image Generationcited by 1
  11. 2026VibeToken: Scaling 1D Image Tokenizers and Autoregressive Models for Dynamic Resolution Generationscited by 1
  12. 2026RankE: End-to-End Post-Training for Discrete Text-to-Image Generation with Decoder Co-Evolutioncited by 1
  13. 2026ThinkingVLA: Interleaved Vision and Language Reasoning for Robotic Manipulationcited by 1
  14. 2026ViQ: Text-Aligned Visual Quantized Representations at Any Resolutioncited by 1
  15. 2026GEAR: Guided End-to-End AutoRegression for Image Synthesiscited by 1
  16. 2026Beyond Token-Level Cross-Entropy: Fr'echet Distributional Post-Training for Autoregressive Image Generationcited by 1
  17. 2026VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervisioncited by 1
  18. 2026UniWeTok: An Unified Binary Tokenizer with Codebook Size mathit2128 for Unified Multimodal Large Language Modelcited by 0
  19. 2026LGQ: Learnable Geometric Quantization for Image Tokenizationcited by 0
  20. 2026Autoregressive Visual Generation Needs a Prologuecited by 0
  21. 2026Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generationcited by 0
  22. 2026InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generationcited by 0
  23. 2026Channel-wise Vector Quantizationcited by 0
  24. 2026MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Mergingcited by 0
  25. 2026Diffusion Image Generation with Explicit Modeling of Data Manifold Geometrycited by 0
  26. 2026Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesiscited by 0
  27. 2026dRAE: Representation Autoencoder with Hyper-Spherical Codescited by 0
  28. 2026Learning to Tessellate: Point Cloud Generation via Recursive Spectral Partitioningcited by 0
  29. 2026Studying Image Tokenizers as Visual Languages in Unified Multimodal Modelscited by 0
  30. 2026StableVQ: Practical Guidelines for Stable Vector-Quantized Tokenizer Trainingcited by 0
  31. 2026NesTok: Nested Self-Aligned 1D Tokenizer for Autoregressive Image Generationcited by 0
  32. 2026Gestalt: Large Multimodal Interplay Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers
  2. 2024Factorized Visual Tokenization and Generation
  3. 2025VAEVQ: Enhancing Discrete Visual Tokenization through Variational Modeling
  4. 2025MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantization
  5. 2025Scalable Training for Vector-Quantized Networks with 100% Codebook Utilization
  6. 2026A Unified Rate-Distortion Perspective on Vector, Product, and Scalar Quantization
  7. 2025Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation
  8. 2023Finite Scalar Quantization: VQ-VAE Made Simple