mlsanityData to 7 Oct 2026

UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding

arXiv 2504.04423 ↗First posted 6 Apr 2025cs.CV cs.AIcited by 49

Not on the map. Why?

Abstract (arXiv)

We introduce UniToken, an auto-regressive generation model that encodes visual inputs through a combination of discrete and continuous representations, enabling seamless integration of unified visual understanding and image generation tasks. Unlike previous approaches that rely on unilateral visual representations, our unified visual encoding framework captures both high-level semantics and low-level details, delivering multidimensional information that empowers heterogeneous tasks to selectively assimilate domain-specific knowledge based on their inherent characteristics. Through in-depth experiments, we uncover key principles for developing a unified model capable of both visual understanding and image generation. Extensive evaluations across a diverse range of prominent benchmarks demonstrate that UniToken achieves state-of-the-art performance, surpassing existing approaches. These results establish UniToken as a robust foundation for future research in this domain. The code and models are available at https://github.com/SxJyJay/UniToken.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 63 of its 72 references that are in the map. First posted on arXiv 6 Apr 2025.

Builds on

63 of 72 references.

References · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  12. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  13. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  14. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  15. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  16. 2016A Diagram Is Worth A Dozen Imagescited by 793
  17. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  18. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  19. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  20. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  21. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  22. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  23. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  24. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  25. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  26. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  27. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  28. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  29. 2021OCR-free Document Understanding Transformercited by 400
  30. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  31. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  32. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  33. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  34. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  35. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  36. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  37. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  38. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 153

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 29; 2026: 20.

49 papers.

Sort:

By area:

  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  3. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  4. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  5. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
  6. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49
  7. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 35
  8. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  9. 2025AToken: A Unified Tokenizer for Visioncited by 30
  10. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  11. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  12. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 27
37 more
  1. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  2. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  3. 2025STAGE: Stable and Generalizable GRPO for Autoregressive Image Generationcited by 15
  4. 2025Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropycited by 11
  5. 2025HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generationcited by 11
  6. 2026Quantifying the Gap between Understanding and Generation within Unified Multimodal Modelscited by 11
  7. 2025ControlThinker: Unveiling Latent Semantics for Controllable Image Generation through Visual Reasoningcited by 8
  8. 2025Locality-aware Parallel Decoding for Efficient Autoregressive Image Generationcited by 8
  9. 2025UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in Reinforcement Learningcited by 8
  10. 2026Unified Personalized Understanding, Generating and Editingcited by 8
  11. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  12. 2025R-Genie: Reasoning-Guided Generative Image Editingcited by 6
  13. 20252D Gaussian Splatting with Semantic Alignment for Image Inpaintingcited by 4
  14. 2025IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generationcited by 4
  15. 2026UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformercited by 4
  16. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  17. 2026VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latentscited by 3
  18. 2025NeoBabel: A Multilingual Open Tower for Visual Generationcited by 2
  19. 2026HYDRA: Unifying Multi-modal Generation and Understanding via Representation-Harmonized Tokenizationcited by 2
  20. 2025Identity-Aware Vision-Language Model for Explainable Face Forgery Detectioncited by 1
  21. 2025OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Taskscited by 1
  22. 2025PairUni: Pairwise Training for Unified Multimodal Language Modelscited by 1
  23. 2025Learning with Preserving for Continual Multitask Learningcited by 1
  24. 2026AEGIS: Exploring the Limit of World Knowledge Capabilities for Unified Mulitmodal Modelscited by 1
  25. 2026UniCompress: Token Compression for Unified Vision-Language Understanding and Generationcited by 1
  26. 2026Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generatorcited by 1
  27. 2026Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generationcited by 1
  28. 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 1
  29. 2025UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generationcited by 0
  30. 2026How Long Can Unified Multimodal Models Generate Images Reliably? Taming Long-Horizon Interleaved Image Generation via Context Curationcited by 0
  31. 2026SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignmentcited by 0
  32. 2026GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoningcited by 0
  33. 2026MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Mergingcited by 0
  34. 2026Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generationcited by 0
  35. 2026Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steeringcited by 0
  36. 2026Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Modelscited by 0
  37. 2026C3-UniMM: Causal Cycle-Consistent Unified Multimodal Modeling via Super Alignment and Shared Decoding Spacecited by 0