mlsanityData to 14 Oct 2026

Making LLaMA SEE and Draw with SEED Tokenizer

arXiv 2310.01218 ↗First posted 2 Oct 2023cs.CVcited by 185

Not on the map. Why?

Cited by 62 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The great success of Large Language Models (LLMs) has expanded the potential of multimodality, contributing to the gradual evolution of General Artificial Intelligence (AGI). A true AGI agent should not only possess the capability to perform predefined multi-tasks but also exhibit emergent abilities in an open-world context. However, despite the considerable advancements made by recent multimodal LLMs, they still fall short in effectively unifying comprehension and generation tasks, let alone open-world emergent abilities. We contend that the key to overcoming the present impasse lies in enabling text and images to be represented and processed interchangeably within a unified autoregressive Transformer. To this end, we introduce SEED, an elaborate image tokenizer that empowers LLMs with the ability to SEE and Draw at the same time. We identify two crucial design principles: (1) Image tokens should be independent of 2D physical patch positions and instead be produced with a 1D causal dependency, exhibiting intrinsic interdependence that aligns with the left-to-right autoregressive prediction mechanism in LLMs. (2) Image tokens should capture high-level semantics consistent with the degree of semantic abstraction in words, and be optimized for both discriminativeness and reconstruction during the tokenizer training phase. With SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe, i.e., next-word prediction. SEED-LLaMA is therefore produced by large-scale pretraining and instruction tuning on the interleaved textual and visual data, demonstrating impressive performance on a broad range of multimodal comprehension and generation tasks. More importantly, SEED-LLaMA has exhibited compositional emergent abilities such as multi-turn in-context multimodal generation, acting like your AI assistant.

Key facts. Cited by 185 papers in the map (data through 14 Oct 2026). Builds on 59 of its 68 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

59 of 68 references.

References · 59

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2017Neural Discrete Representation Learningcited by 4,712
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  13. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  16. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  17. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  18. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  19. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  20. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  21. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  22. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  23. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  24. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  25. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  26. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  27. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  28. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  29. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  30. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  31. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 379
  32. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  33. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  34. 2022Visual Spatial Reasoningcited by 293
  35. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  36. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  37. 2023Generating Images with Multimodal Language Modelscited by 260
  38. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 82; 2025: 57; 2026: 35.

185 papers.

Sort:

By area:

  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  2. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  3. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  4. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  5. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  6. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  7. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  8. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  9. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  10. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  11. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 177
  12. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
38 more
  1. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  2. 2024In-Context LoRA for Diffusion Transformerscited by 158
  3. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  4. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  5. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  6. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  7. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 120
  8. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 118
  9. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 109
  10. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  11. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 98
  12. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
  13. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  14. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  15. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  16. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  17. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 73
  18. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 69
  19. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhancecited by 64
  20. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 56
  21. 2024Reconstructive Visual Instruction Tuningcited by 55
  22. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  23. 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Spacecited by 51
  24. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  25. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  26. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 44
  27. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44
  28. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
  29. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  30. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  31. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 41
  32. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 41
  33. 2025ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoningcited by 37
  34. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 36
  35. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  36. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  37. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  38. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 34

50 most cited of 185 All 185 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM-I: LLMs are Naturally Interleaved Multimodal Creators
  2. 2026Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
  3. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning
  4. 2025Improving Chain-of-Thought Efficiency for Autoregressive Image Generation
  5. 2026Token-Efficient Multimodal Reasoning via Image Prompt Packaging
  6. 2026ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation
  7. 2026SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning
  8. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks