mlsanityData to 14 Oct 2026

MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding

arXiv 2411.17762 ↗First posted 26 Nov 2024cs.CVcited by 44

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce MUSE-VL, a Unified Vision-Language Model through Semantic discrete Encoding for multimodal understanding and generation. Recently, the research community has begun exploring unified models for visual generation and understanding. However, existing vision tokenizers (e.g., VQGAN) only consider low-level information, which makes it difficult to align with language tokens. This results in high training complexity and necessitates a large amount of training data to achieve optimal performance. Additionally, their performance is still far from dedicated understanding models. This paper proposes Semantic Discrete Encoding (SDE), which effectively aligns the information of visual tokens and language tokens by adding semantic constraints to the visual tokenizer. This greatly reduces the amount of training data and improves the performance of the unified model. With the same LLM size, our method improved the understanding performance by 4.8% compared to the previous SOTA Emu3 and surpassed the dedicated understanding model LLaVA-NeXT 34B by 3.7%. Our model also surpasses the existing unified models on visual generation benchmarks.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 69 of its 85 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

69 of 85 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  2. 2024SOLO: A Single Transformer for Scalable Vision-Language Modeling8 Jul
  3. 2024Unveiling Encoder-Free Vision-Language Models17 Jun
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
38 more
  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2024Mixtral of Expertscited by 1,823
  12. 2019Towards VQA Models That Can Readcited by 1,718
  13. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  14. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  15. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  16. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  17. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  18. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  19. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  20. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  21. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  22. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  23. 2016A Diagram Is Worth A Dozen Imagescited by 804
  24. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  25. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  26. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  27. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  28. 2024Yi: Open Foundation Models by 01.AIcited by 719
  29. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  30. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  31. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  32. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  33. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  34. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  35. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  36. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  37. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  38. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 19.

44 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  2. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  3. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  4. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  5. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  6. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  7. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 41
  8. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 36
  9. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  10. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  11. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generationcited by 21
  12. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
32 more
  1. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generationcited by 18
  2. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  3. 2025Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Bettercited by 17
  4. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  5. 2025VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstructioncited by 10
  6. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  7. 2025EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecturecited by 8
  8. 2025Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLMcited by 7
  9. 2025End-to-End Vision Tokenizer Tuningcited by 5
  10. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  11. 2026ChatUMM: Robust Context Tracking for Conversational Interleaved Generationcited by 3
  12. 2025Aquarius: A Family of Industry-Level Video Generation Models for Marketing Scenarioscited by 2
  13. 2026HYDRA: Representation Harmonized Tokenization for Multimodal Generation and Understandingcited by 2
  14. 2026HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizerscited by 2
  15. 2025MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Modelscited by 1
  16. 2025UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perceptioncited by 1
  17. 2026UniCompress: Token Compression for Unified Vision-Language Understanding and Generationcited by 1
  18. 2026LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generationcited by 1
  19. 2026Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusioncited by 1
  20. 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokenscited by 1
  21. 2026ViQ: Text-Aligned Visual Quantized Representations at Any Resolutioncited by 1
  22. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 1
  23. 2025UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generationcited by 0
  24. 2026InfoTok: Information-Theoretic Regularization for Capacity-Constrained Shared Visual Tokenization in Unified MLLMscited by 0
  25. 2026Decoupling Vision and Language: Codebook Anchored Visual Adaptationcited by 0
  26. 2026UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representationscited by 0
  27. 2026Unified Pix Token And Word Token Generative Language Modelcited by 0
  28. 2026Channel-wise Vector Quantizationcited by 0
  29. 2026Bridging Video Understanding and Generation in a Unified Frameworkcited by 0
  30. 2026WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modelingcited by 0
  31. 2026dRAE: Representation Autoencoder with Hyper-Spherical Codescited by 0
  32. 2026C3-UniMM: Causal Cycle-Consistent Unified Multimodal Modeling via Super Alignment and Shared Decoding Spacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024An Introduction to Vision-Language Modeling
  2. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  3. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding
  4. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
  5. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Models
  6. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  7. 2025Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Study
  8. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations