mlsanityData to 7 Oct 2026

UniCode: Learning a Unified Codebook for Multimodal Large Language Models

arXiv 2403.09072 ↗First posted 14 Mar 2024cs.CV cs.AI cs.CLcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In this paper, we propose UniCode, a novel approach within the domain of multimodal large language models (MLLMs) that learns a unified codebook to efficiently tokenize visual, text, and potentially other types of signals. This innovation addresses a critical limitation in existing MLLMs: their reliance on a text-only codebook, which restricts MLLM's ability to generate images and texts in a multimodal context. Towards this end, we propose a language-driven iterative training paradigm, coupled with an in-context pre-training task we term ``image decompression'', enabling our model to interpret compressed visual data and generate high-quality images.The unified codebook empowers our model to extend visual instruction tuning to non-linguistic generation tasks. Moreover, UniCode is adaptable to diverse stacked quantization approaches in order to compress visual signals into a more compact token representation. Despite using significantly fewer parameters and less data during training, Unicode demonstrates promising capabilities in visual reconstruction and generation. It also achieves performances comparable to leading MLLMs across a spectrum of VQA benchmarks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 65 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

54 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  11. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  12. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  14. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  15. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  16. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  17. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  18. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  19. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  20. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  21. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  22. 2020Jukebox: A Generative Model for Musiccited by 535
  23. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  24. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  25. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  26. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  27. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  28. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  29. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  30. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  31. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  32. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  33. 2023SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMscited by 54
  34. 2023Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worldscited by 45
  35. 2022Temporally Consistent Transformers for Video Generationcited by 43
  36. 2017Perceptual Adversarial Networks for Image-to-Image Transformationcited by 42
  37. 2014Stacked Quantizers for Compositional Vector Compressioncited by 30
  38. 2023Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignmentcited by 28

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 2.

14 papers.

Sort:

By area:

  1. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  2. 2024LARP: Tokenizing Videos with a Learned Autoregressive Generative Priorcited by 46
  3. 2024The Evolution of Multimodal Model Architecturescited by 21
  4. 2024Scaling Large Motion Models with Million-Level Human Motionscited by 20
  5. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalitiescited by 16
  6. 2024Node Identifiers: Compact, Discrete Representations for Efficient Graph Learningcited by 10
  7. 2025LangBridge: Interpreting Image as a Combination of Language Embeddingscited by 9
  8. 2025Unified Multimodal Understanding via Byte-Pair Visual Encodingcited by 9
  9. 2025Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learningcited by 5
  10. 2024QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worldscited by 4
  11. 2024ARCON: Advancing Auto-Regressive Continuation for Driving Videoscited by 1
  12. 2025Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentationcited by 1
2 more
  1. 2026LooC: Effective Low-Dimensional Codebook for Compositional Vector Quantizationcited by 0
  2. 2026Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Featurescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation
  2. 2026UniWeTok: An Unified Binary Tokenizer with Codebook Size mathit2128 for Unified Multimodal Large Language Model
  3. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  4. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
  5. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
  6. 2024FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression
  7. 2026A More Word-like Image Tokenization for MLLMs
  8. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models