mlsanityData to 7 Oct 2026

Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization

arXiv 2309.04669 ↗First posted 9 Sep 2023cs.CVcited by 87

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Recently, the remarkable advance of the Large Language Model (LLM) has inspired researchers to transfer its extraordinary reasoning capability to both vision and language data. However, the prevailing approaches primarily regard the visual input as a prompt and focus exclusively on optimizing the text generation process conditioned upon vision content by a frozen LLM. Such an inequitable treatment of vision and language heavily constrains the model's potential. In this paper, we break through this limitation by representing both vision and language in a unified form. Specifically, we introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language that LLM can read. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. Coped with this tokenizer, the presented foundation model called LaVIT can handle both image and text indiscriminately under the same generative learning paradigm. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously. Extensive experiments further showcase that it outperforms the existing models by a large margin on massive vision-language tasks. Our code and models are available at https://github.com/jy0205/LaVIT.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 58 references that are in the map. First posted on arXiv 9 Sep 2023.

Builds on

50 of 58 references.

References · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  13. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  14. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  15. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  16. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  17. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  18. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  19. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  20. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  21. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  22. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  23. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  24. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  25. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  26. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  27. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  28. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  29. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  30. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  31. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  32. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  33. 2018nocaps: novel object captioning at scalecited by 444
  34. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  35. 2023Generating Images with Multimodal Language Modelscited by 259
  36. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  37. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  38. 2022Language Models are General-Purpose Interfacescited by 96

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 26; 2025: 41; 2026: 17.

87 papers.

Sort:

By area:

  1. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  2. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  3. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  4. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  5. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  6. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  7. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  8. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  9. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  10. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  11. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  12. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 68
38 more
  1. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 53
  2. 2025Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructionscited by 50
  3. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  4. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  5. 2024MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesiscited by 44
  6. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44
  7. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  8. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  9. 2024Autoregressive Models in Vision: A Surveycited by 39
  10. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  11. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  13. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 31
  14. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  15. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokenscited by 27
  16. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generationcited by 21
  17. 2025Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMscited by 20
  18. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  19. 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
  20. 2025Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inferencecited by 18
  21. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  22. 2024MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotationscited by 14
  23. 2024When Worse is Better: Navigating the compression-generation tradeoff in visual tokenizationcited by 14
  24. 2024Probing Multimodal Large Language Models for Global and Local Semantic Representationscited by 12
  25. 2024TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Modelscited by 12
  26. 2024SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretizationcited by 12
  27. 2025OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Modelscited by 11
  28. 2025UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generationcited by 11
  29. 2026Quantifying the Gap between Understanding and Generation within Unified Multimodal Modelscited by 11
  30. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  31. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalistscited by 9
  32. 2024UniRAG: Universal Retrieval Augmentation for Large Vision Language Modelscited by 8
  33. 2025RAGAR: Retrieval Augmented Personalized Image Generation Guided by Recommendationcited by 8
  34. 2025AUGUSTUS: An LLM-Driven Multimodal Agent System with Contextualized User Memorycited by 8
  35. 2025Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLMcited by 7
  36. 2025Common Inpainted Objects In-N-Out of Contextcited by 7
  37. 2025Quantize-then-Rectify: Efficient VQ-VAE Trainingcited by 7
  38. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Planting a SEED of Vision in Large Language Model
  2. 2025Vision-Enhanced Large Language Models for High-Resolution Image Synthesis and Multimodal Data Interpretation
  3. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
  4. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
  5. 2023What Makes for Good Visual Tokenizers for Large Language Models?
  6. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs
  7. 2026UniWeTok: An Unified Binary Tokenizer with Codebook Size mathit2128 for Unified Multimodal Large Language Model
  8. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See