mlsanityData to 14 Oct 2026

UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation

arXiv 2506.20214 ↗First posted 25 Jun 2025cs.CV cs.MMcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Unified multimodal large language models (MLLMs) have shown promise in jointly advancing multimodal understanding and generation, with visual codebooks discretizing images into tokens for autoregressive modeling. Existing codebook-based methods either rely on small vocabularies (~16K entries) that lack fine-grained semantics or naively scale up, resulting in low token utilization and unstable training. We propose UniCode2, a cascaded codebook framework enabling large-scale, semantically aligned, and stable visual tokenization. By clustering millions of SigLIP sequence embeddings, we build a 500K-entry codebook that preserves vision-language alignment while expanding capacity. Stability is ensured via a cascaded design: a frozen codebook anchors the embedding space, and a trainable codebook refines task-specific semantics. This decoupling promotes high utilization and robust learning. Moreover, the alignment of our visual tokens with textual semantics enables seamless integration with pretrained diffusion decoders, supporting high-quality visual synthesis with minimal adaptation. UniCode2 delivers strong performance across diverse benchmarks, demonstrating the viability of scaling visual token spaces without sacrificing stability, semantics, or modularity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 72 of its 87 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

72 of 87 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance9 Dec
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  5. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation8 Jul
  6. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Model31 May
  7. 2023Generative Multimodal Models are In-Context Learners20 Dec
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  9. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  10. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  11. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
3 more
  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023JourneyDB: A Benchmark for Generative Image Understanding3 Jul

Other references · 57

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
  1. 2019Towards VQA Models That Can Readcited by 1,718
  2. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  3. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  4. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  6. 2016A Diagram Is Worth A Dozen Imagescited by 804
  7. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  8. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  9. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  10. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  12. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  13. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  14. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  15. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  16. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  17. 2024OmniGen: Unified Image Generationcited by 394
  18. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  19. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  20. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  21. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  22. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  23. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  24. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  25. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  26. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 188
  27. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  28. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 118
  29. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  30. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 80
  31. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  32. 2025Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generationcited by 71
  33. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68
  34. 2023Online Clustered Codebookcited by 55
  35. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 54
  36. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  37. 2025QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generationcited by 45
  38. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024UniCode: Learning a Unified Codebook for Multimodal Large Language Models
  2. 2026Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation
  3. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  4. 2026UniWeTok: An Unified Binary Tokenizer with Codebook Size mathit2128 for Unified Multimodal Large Language Model
  5. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
  6. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
  7. 2026A More Word-like Image Tokenization for MLLMs
  8. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs