mlsanityData to 14 Oct 2026

Image Understanding Makes for A Good Tokenizer for Image Generation

arXiv 2411.04406 ↗First posted 7 Nov 2024cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

Abstract Modern image generation (IG) models have been shown to capture rich semantics valuable for image understanding (IU) tasks. However, the potential of IU models to improve IG performance remains uncharted. We address this issue using a token-based IG framework, which relies on effective tokenizers to project images into token sequences. Currently, pixel reconstruction (e.g., VQGAN) dominates the training objective for image tokenizers. In contrast, our approach adopts the feature reconstruction objective, where tokenizers are trained by distilling knowledge from pretrained IU encoders. Comprehensive comparisons indicate that tokenizers with strong IU capabilities achieve superior IG performance across a variety of metrics, datasets, tasks, and proposal networks. Notably, VQ-KD CLIP achieves 4.10 FID on ImageNet-1k (IN-1k). Visualization suggests that the superiority of VQ-KD can be partly attributed to the rich semantics within the VQ-KD codebook. We further introduce a straightforward pipeline to directly transform IU encoders into tokenizers, demonstrating exceptional effectiveness for IG tasks. These discoveries may energize further exploration into image tokenizer research and inspire the community to reassess the relationship between IU and IG. The code is released at https://github.com/magic-research/vectorquantization.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 44 of its 55 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

44 of 55 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
32 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  10. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  11. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  12. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  13. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  14. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  15. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 449
  16. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  17. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  18. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  19. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  20. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 170
  21. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125
  22. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 123
  23. 2023SODA: Bottleneck Diffusion Models for Representation Learningcited by 77
  24. 2022SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantizationcited by 68
  25. 2023FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Modelscited by 61
  26. 2023Diffusion Models Beat GANs on Image Classificationcited by 58
  27. 2023Online Clustered Codebookcited by 55
  28. 2022Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Modelscited by 54
  29. 2022Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesiscited by 31
  30. 2023HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayescited by 27
  31. 2023DiffusionEngine: Diffusion Model is Scalable Data Engine for Object Detectioncited by 23
  32. 2023Vector Quantized Wasserstein Auto-Encodercited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 57
  2. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  3. 2024When Worse is Better: Navigating the compression-generation tradeoff in visual tokenizationcited by 15
  4. 2026Making Reconstruction FID Predictive of Diffusion Generation FIDcited by 13
  5. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantizationcited by 12
  6. 2024Factorized Visual Tokenization and Generationcited by 11
  7. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  8. 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 3
  9. 2026MacTok: Robust Continuous Tokenization for Image Generationcited by 2
  10. 2025Training-Free Vector Quantization via Gaussian VAEscited by 1
  11. 2025UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generationcited by 0
  12. 2025Codebook-Based Adaptive Feature Compression With Semantic Enhancement for Edge-Cloud Systemscited by 0
3 more
  1. 2026UniMark: Unified Adaptive Multi-bit Watermarking for Autoregressive Image Generatorscited by 0
  2. 2026Generative Lane Topology Reasoning via Autoregressive Model with Geometry Priorcited by 0
  3. 2026From Pixels, Without Pre-training: Joint Generative and Self-Supervised Representation Learning in One Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation
  2. 2025UniTok: A Unified Tokenizer for Visual Generation and Understanding
  3. 2026MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
  4. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation
  5. 2026Semantic-Aware Prefix Learning for Token-Efficient Image Generation
  6. 2024An Image is Worth 32 Tokens for Reconstruction and Generation
  7. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement
  8. 2025Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation