mlsanityData to 7 Oct 2026

TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation

arXiv 2412.03069 ↗First posted 4 Dec 2024cs.CV cs.AIcited by 200

Not on the map. Why?

Cited by 34 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present TokenFlow, a novel unified image tokenizer that bridges the long-standing gap between multimodal understanding and generation. Prior research attempt to employ a single reconstruction-targeted Vector Quantization (VQ) encoder for unifying these two tasks. We observe that understanding and generation require fundamentally different granularities of visual information. This leads to a critical trade-off, particularly compromising performance in multimodal understanding tasks. TokenFlow addresses this challenge through an innovative dual-codebook architecture that decouples semantic and pixel-level feature learning while maintaining their alignment via a shared mapping mechanism. This design enables direct access to both high-level semantic representations crucial for understanding tasks and fine-grained visual features essential for generation through shared indices. Our extensive experiments demonstrate TokenFlow's superiority across multiple dimensions. Leveraging TokenFlow, we demonstrate for the first time that discrete visual input can surpass LLaVA-1.5 13B in understanding performance, achieving a 7.2% average improvement. For image reconstruction, we achieve a strong FID score of 0.63 at 384*384 resolution. Moreover, TokenFlow establishes state-of-the-art performance in autoregressive image generation with a GenEval score of 0.55 at 256*256 resolution, achieving comparable results to SDXL.

Key facts. Cited by 200 papers in the map (data through 7 Oct 2026). Builds on 60 of its 75 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

60 of 75 references.

References · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  13. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  14. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  15. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  16. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  17. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  18. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  19. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  20. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  21. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  22. 2016A Diagram Is Worth A Dozen Imagescited by 793
  23. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  24. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  25. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  26. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  27. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  28. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  29. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  30. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  31. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  32. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  33. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  34. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  35. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  36. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  37. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  38. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 105; 2026: 92.

200 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  3. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  4. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  5. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  6. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  7. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  8. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  9. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  10. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  11. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 93
  12. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
38 more
  1. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  2. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  3. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  4. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 62
  5. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  6. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 57
  7. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 55
  8. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 53
  9. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  10. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
  11. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  12. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  13. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 45
  14. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 43
  15. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  16. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 40
  17. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 38
  18. 2025ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoningcited by 37
  19. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 35
  20. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  21. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  22. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  23. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 31
  24. 2025MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPOcited by 30
  25. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 27
  26. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  27. 2024Next Patch Prediction for Autoregressive Visual Generationcited by 25
  28. 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 23
  29. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  30. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generationcited by 21
  31. 2025The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencodingcited by 21
  32. 2025Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generationcited by 20
  33. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  34. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generationcited by 17
  35. 2025DetailFlow: 1D Coarse-to-Fine Autoregressive Image Generation via Next-Detail Predictioncited by 17
  36. 2025Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Bettercited by 17
  37. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
  38. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 17

50 most cited of 200 All 200 (JSON)