mlsanityData to 7 Oct 2026

UniTok: A Unified Tokenizer for Visual Generation and Understanding

arXiv 2502.20321 ↗First posted 27 Feb 2025cs.CV cs.AIcited by 148

Not on the map. Why?

Cited by 27 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Visual generative and understanding models typically rely on distinct tokenizers to process images, presenting a key challenge for unifying them within a single framework. Recent studies attempt to address this by connecting the training of VQVAE (for autoregressive generation) and CLIP (for understanding) to build a unified tokenizer. However, directly combining these training objectives has been observed to cause severe loss conflicts. In this paper, we show that reconstruction and semantic supervision do not inherently conflict. Instead, the underlying bottleneck stems from limited representational capacity of discrete token space. Building on these insights, we introduce UniTok, a unified tokenizer featuring a novel multi-codebook quantization mechanism that effectively scales up the vocabulary size and bottleneck dimension. In terms of final performance, UniTok sets a new record of 0.38 rFID and 78.6% zero-shot accuracy on ImageNet. Besides, UniTok can be seamlessly integrated into MLLMs to unlock native visual generation capability, without compromising the understanding performance. Additionally, we show that UniTok favors cfg-free generation, reducing gFID from 14.6 to 2.5 on ImageNet 256×256 benchmark. GitHub: https://github.com/FoundationVision/UniTok.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Builds on 67 of its 80 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

67 of 80 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  12. 2022Scalable Diffusion Models with Transformerscited by 5,931
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  10. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  11. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  12. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  13. 2023A Survey on Multimodal Large Language Modelscited by 898
  14. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  15. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  16. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  17. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  18. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  19. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  20. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  21. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  22. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  23. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  24. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  25. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  26. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  27. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  28. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  29. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  30. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  31. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  32. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  33. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376
  34. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  35. 2022MAGVIT: Masked Generative Video Transformercited by 341
  36. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  37. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  38. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 69; 2026: 79.

148 papers.

Sort:

By area:

  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  2. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  3. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  4. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  5. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  6. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 57
  7. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 55
  8. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
  9. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  10. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 43
  11. 2025InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generationcited by 43
  12. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
38 more
  1. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generationcited by 42
  2. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 40
  3. 2025JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agentcited by 39
  4. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 35
  5. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  6. 2025EO-1: An Open Unified Embodied Foundation Model for General Robot Controlcited by 30
  7. 2025AToken: A Unified Tokenizer for Visioncited by 30
  8. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 27
  9. 2025One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generationcited by 27
  10. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  11. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  12. 2025Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editingcited by 22
  13. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generationcited by 21
  14. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  15. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 17
  16. 2025VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Modelscited by 16
  17. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  18. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizercited by 15
  19. 2025STAGE: Stable and Generalizable GRPO for Autoregressive Image Generationcited by 15
  20. 2025TokBench: Evaluating Your Visual Tokenizer before Visual Generationcited by 12
  21. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  22. 2025GeoUni: A Unified Model for Generating Geometry Diagrams, Problems and Problem Solutionscited by 11
  23. 2025Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropycited by 11
  24. 2026Quantifying the Gap between Understanding and Generation within Unified Multimodal Modelscited by 11
  25. 2025Scalable Training for Vector-Quantized Networks with 100% Codebook Utilizationcited by 10
  26. 2025MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantizationcited by 9
  27. 2025WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstructioncited by 9
  28. 2025VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstructioncited by 9
  29. 2026ERNIE 5.0 Technical Reportcited by 9
  30. 2025Locality-aware Parallel Decoding for Efficient Autoregressive Image Generationcited by 8
  31. 2025UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in Reinforcement Learningcited by 8
  32. 2026Unified Personalized Understanding, Generating and Editingcited by 8
  33. 2026Autoregressive Image Generation with Masked Bit Modelingcited by 8
  34. 2025Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLMcited by 7
  35. 2025Make It Efficient: Dynamic Sparse Attention for Autoregressive Image Generationcited by 7
  36. 2025Quantize-then-Rectify: Efficient VQ-VAE Trainingcited by 7
  37. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  38. 2025Semantic Context Matters: Improving Conditioning for Autoregressive Modelscited by 7

50 most cited of 148 All 148 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding
  2. 2025V2Flow: Unifying Visual Tokenization and Large Language Model Vocabularies for Autoregressive Image Generation
  3. 2026Semantic-Aware Prefix Learning for Token-Efficient Image Generation
  4. 2025Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
  5. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
  6. 2026G2TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models
  7. 2024Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformers
  8. 2025Adapting Self-Supervised Representations as a Latent Space for Efficient Generation