# DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies

arXiv: https://arxiv.org/abs/2503.14324 · HTML: https://mlsanity.com/paper/2503.14324/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 80 of its 101 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Builds on (most cited first, 30 of 80)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)

## Built on by (most cited first, 30 of 42)

- [Show-o2: Improved Native Unified Multimodal Models](https://mlsanity.com/paper/2506.15564/) (2025)
- [T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT](https://mlsanity.com/paper/2505.00703/) (2025)
- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation](https://mlsanity.com/paper/2504.02782/) (2025)
- [AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models](https://mlsanity.com/paper/2509.25162/) (2025)
- [TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models](https://mlsanity.com/paper/2512.02014/) (2025)
- [SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture](https://mlsanity.com/paper/2605.12500/) (2026)
- [A Survey of Token Compression for Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2507.20198/) (2025)
- [UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation](https://mlsanity.com/paper/2510.10575/) (2025)
- [Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing](https://mlsanity.com/paper/2512.17909/) (2025)
- [Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation](https://mlsanity.com/paper/2507.08441/) (2025)
- [Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better](https://mlsanity.com/paper/2506.09040/) (2025)
- [Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation](https://mlsanity.com/paper/2507.01957/) (2025)
- [VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction](https://mlsanity.com/paper/2511.23386/) (2025)
- [Scone: Bridging Composition and Distinction in Subject-Driven Image Generation via Unified Understanding-Generation Modeling](https://mlsanity.com/paper/2512.12675/) (2025)
- [WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction](https://mlsanity.com/paper/2508.05599/) (2025)
- [LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model](https://mlsanity.com/paper/2604.02097/) (2026)
- [CaTok: Taming Mean Flows for One-Dimensional Causal Image Tokenization](https://mlsanity.com/paper/2603.06449/) (2026)
- [UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer](https://mlsanity.com/paper/2606.16255/) (2026)
- [UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2511.16917/) (2025)
- [NeoBabel: A Multilingual Open Tower for Visual Generation](https://mlsanity.com/paper/2507.06137/) (2025)
- [Visual Generation Tuning](https://mlsanity.com/paper/2511.23469/) (2025)
- [Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification](https://mlsanity.com/paper/2601.20742/) (2026)
- [EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generation](https://mlsanity.com/paper/2603.12108/) (2026)
- [DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders](https://mlsanity.com/paper/2605.22777/) (2026)
- [IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder](https://mlsanity.com/paper/2606.11096/) (2026)
- [HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers](https://mlsanity.com/paper/2606.13289/) (2026)
- [Kelix Technical Report](https://mlsanity.com/paper/2602.09843/) (2026)
- [WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens](https://mlsanity.com/paper/2605.18115/) (2026)
- [Vision Foundation Models as Generalist Tokenizers for Image Generation](https://mlsanity.com/paper/2605.18390/) (2026)

## Abstract (arXiv)

The differing representation spaces required for visual understanding and generation pose a challenge in unifying them within the autoregressive paradigm of large language models. A vision tokenizer trained for reconstruction excels at capturing low-level visual appearance, making it well-suited for visual generation but lacking high-level semantic representations for understanding tasks. Conversely, a vision encoder trained via contrastive learning aligns well with language but struggles to decode back into the pixel space for generation tasks. To bridge this gap, we propose DualToken, a method that unifies representations for both understanding and generation within a single tokenizer. However, directly integrating reconstruction and semantic objectives creates conflicts, leading to degraded performance in both reconstruction fidelity and semantic accuracy. Instead of forcing a single codebook to capture both visual appearance and semantics, DualToken disentangles them by introducing separate codebooks for high-level semantics and low-level visual details. As a result, DualToken achieves 0.25 rFID and 82.0% zero-shot accuracy on ImageNet, and demonstrates strong effectiveness in downstream MLLM tasks for both understanding and generation. Specifically, our method surpasses VILA-U by 5.8 points on average across ten visual understanding benchmarks and delivers a 13% improvement on GenAI-Bench. Notably, incorporating dual visual tokens outperforms using a single token type on both understanding and generation tasks. We hope our research offers a new perspective on leveraging dual visual vocabularies for building unified vision-language models. Project page is available at https://songweii.github.io/dualtoken-project-page.