mlsanityData to 14 Oct 2026

Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations

arXiv 2506.18898 ↗First posted 23 Jun 2025cs.CV cs.AI cs.CLcited by 44

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This paper presents a multimodal framework that attempts to unify visual understanding and generation within a shared discrete semantic representation. At its core is the Text-Aligned Tokenizer (TA-Tok), which converts images into discrete tokens using a text-aligned codebook projected from a large language model's (LLM) vocabulary. By integrating vision and text into a unified space with an expanded vocabulary, our multimodal LLM, Tar, enables cross-modal input and output through a shared interface, without the need for modality-specific designs. Additionally, we propose scale-adaptive encoding and decoding to balance efficiency and visual detail, along with a generative de-tokenizer to produce high-fidelity visual outputs. To address diverse decoding needs, we utilize two complementary de-tokenizers: a fast autoregressive model and a diffusion-based model. To enhance modality fusion, we investigate advanced pre-training tasks, demonstrating improvements in both visual understanding and generation. Experiments across benchmarks show that Tar matches or surpasses existing multimodal LLM methods, achieving faster convergence and greater training efficiency. Code, models, and data are available at https://tar.csuhan.com

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 80 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

61 of 80 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance9 Dec
  2. 2023Generative Multimodal Models are In-Context Learners20 Dec
  3. 2023OneLLM: One Framework to Align All Modalities with Language6 Dec
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  8. 2023JourneyDB: A Benchmark for Generative Image Understanding3 Jul
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2025Qwen2.5-VL Technical Reportcited by 5,587
  10. 2024Qwen2.5 Technical Reportcited by 4,942
  11. 2017Neural Discrete Representation Learningcited by 4,712
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
38 more
  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  8. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  9. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  10. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  11. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  13. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  14. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  15. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  16. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  17. 2021ImageNet-21K Pretraining for the Massescited by 613
  18. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  19. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  20. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  21. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  22. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  23. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  24. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  25. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  26. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 256
  27. 2025Transfer between Modalities with MetaQueriescited by 236
  28. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  29. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  30. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 174
  31. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  32. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  33. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 104
  34. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
  35. 2025Seaweed-7B: Cost-Effective Training of Video Generation Foundation Modelcited by 82
  36. 2024Dual Diffusion for Unified Image Generation and Understandingcited by 76
  37. 2024LARP: Tokenizing Videos with a Learned Autoregressive Generative Priorcited by 46
  38. 2025QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generationcited by 45

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 30.

44 papers.

Sort:

By area:

  1. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 54
  2. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  3. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 44
  4. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 36
  5. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  6. 2025Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editingcited by 22
  7. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  8. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  9. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizercited by 15
  10. 2025TBAC-UniImage: Unified Understanding and Generation by Ladder-Side Diffusion Tuningcited by 10
  11. 2025VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstructioncited by 10
  12. 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Modelcited by 8
32 more
  1. 2026Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Modelscited by 5
  2. 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modelingcited by 5
  3. 2025Growing Visual Generative Capacity for Pre-Trained MLLMscited by 4
  4. 2026HyperCLOVA X 8B Omnicited by 4
  5. 2026iFSQ: Improving FSQ for Image Generation with 1 Line of Codecited by 4
  6. 2026BitDance: Scaling Autoregressive Generative Models with Binary Tokenscited by 4
  7. 2026Baton: Explicit Semantic Blueprints for Joint Video-Audio Generationcited by 4
  8. 2026Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unificationcited by 4
  9. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  10. 2025Plan-X: Instruct Video Generation via Semantic Planningcited by 3
  11. 2026Multi-Scale Local Speculative Decoding for Image Generationcited by 3
  12. 2026Bridging Information Asymmetry: A Hierarchical Framework for Deterministic Blind Face Restorationcited by 3
  13. 2026Generative Refinement Networks for Visual Synthesiscited by 3
  14. 2026MUSE: Resolving Manifold Misalignment in Visual Tokenization via Topological Orthogonalitycited by 3
  15. 2026Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Renderingcited by 3
  16. 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 3
  17. 2025Mixture of States: Routing Token-Level Dynamics for Multimodal Generationcited by 2
  18. 2026Extending One-Step Image Generation from Class Labels to Text via Discriminative Text Representationcited by 2
  19. 2026Kelix Technical Reportcited by 1
  20. 2026Large Language Models are Universal Reasoners for Visual Generationcited by 1
  21. 2026ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representationscited by 1
  22. 2026SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Modelscited by 1
  23. 2026GEAR: Guided End-to-End AutoRegression for Image Synthesiscited by 1
  24. 2026Twins: Learn to Predict Unified Representations with Focal Losscited by 1
  25. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 1
  26. 2026InfoTok: Information-Theoretic Regularization for Capacity-Constrained Shared Visual Tokenization in Unified MLLMscited by 0
  27. 2026UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representationscited by 0
  28. 2026Does Unification Come at a Cost? Uni-SafeBench: A Safety Benchmark for Unified Multimodal Large Modelscited by 0
  29. 2026Bridging Video Understanding and Generation in a Unified Frameworkcited by 0
  30. 2026dRAE: Representation Autoencoder with Hyper-Spherical Codescited by 0
  31. 2026Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generationcited by 0
  32. 2026Studying Image Tokenizers as Visual Languages in Unified Multimodal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  2. 2023Planting a SEED of Vision in Large Language Model
  3. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
  4. 2025UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation
  5. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer
  6. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation
  7. 2026HYDRA: Representation Harmonized Tokenization for Multimodal Generation and Understanding
  8. 2026Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation