mlsanityData to 7 Oct 2026

Planting a SEED of Vision in Large Language Model

arXiv 2307.08041 ↗First posted 16 Jul 2023cs.CVcited by 140

Not on the map. Why?

Cited by 54 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present SEED, an elaborate image tokenizer that empowers Large Language Models (LLMs) with the emergent ability to SEE and Draw at the same time. Research on image tokenizers has previously reached an impasse, as frameworks employing quantized visual tokens have lost prominence due to subpar performance and convergence in multimodal comprehension (compared to BLIP-2, etc.) or generation (compared to Stable Diffusion, etc.). Despite the limitations, we remain confident in its natural capacity to unify visual and textual representations, facilitating scalable multimodal training with LLM's original recipe. In this study, we identify two crucial principles for the architecture and training of SEED that effectively ease subsequent alignment with LLMs. (1) Image tokens should be independent of 2D physical patch positions and instead be produced with a 1D causal dependency, exhibiting intrinsic interdependence that aligns with the left-to-right autoregressive prediction mechanism in LLMs. (2) Image tokens should capture high-level semantics consistent with the degree of semantic abstraction in words, and be optimized for both discriminativeness and reconstruction during the tokenizer training phase. As a result, the off-the-shelf LLM is able to perform both image-to-text and text-to-image generation by incorporating our SEED through efficient LoRA tuning. Comprehensive multimodal pretraining and instruction tuning, which may yield improved results, are reserved for future investigation. This version of SEED was trained in 5.7 days using only 64 V100 GPUs and 5M publicly available image-text pairs. Our preliminary study emphasizes the great potential of discrete visual tokens in versatile multimodal LLMs and the importance of proper image tokenizers in broader research.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 16 Jul 2023.

Builds on

25 of 31 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
13 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  7. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  8. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  9. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  10. 2018nocaps: novel object captioning at scalecited by 444
  11. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  12. 2023Generating Images with Multimodal Language Modelscited by 259
  13. 2023Emu: Generative Pretraining in Multimodalitycited by 156

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 54; 2025: 42; 2026: 20.

140 papers.

Sort:

By area:

  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  4. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  5. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  6. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  7. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  8. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  9. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  10. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  11. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  12. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
38 more
  1. 2025Transfer between Modalities with MetaQueriescited by 231
  2. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  3. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  4. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  5. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
  6. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  7. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  8. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  9. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  10. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  11. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  12. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  13. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  14. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  15. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  16. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  17. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  18. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  19. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 93
  20. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  21. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  22. 2024Dual Diffusion for Unified Image Generation and Understandingcited by 76
  23. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 70
  24. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 69
  25. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 66
  26. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 62
  27. 2025Scaling Language-Free Visual Representation Learningcited by 62
  28. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  29. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 55
  30. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 55
  31. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  32. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  33. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49
  34. 2023Merlin:Empowering Multimodal LLMs with Foresight Mindscited by 46
  35. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  36. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  37. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44
  38. 2023VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generationcited by 41

50 most cited of 140 All 140 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026A More Word-like Image Tokenization for MLLMs
  2. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
  3. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
  4. 2026What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
  5. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
  6. 2023What Makes for Good Visual Tokenizers for Large Language Models?
  7. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  8. 2026UniWeTok: An Unified Binary Tokenizer with Codebook Size mathit2128 for Unified Multimodal Large Language Model