mlsanityData to 7 Oct 2026

Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning

arXiv 2309.02591 ↗First posted 5 Sep 2023cs.LG cs.CL cs.CVcited by 168

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented, token-based, decoder-only multi-modal language model capable of generating and infilling both text and images. CM3Leon uses the CM3 multi-modal architecture but additionally shows the extreme benefits of scaling up and tuning on more diverse instruction-style data. It is the first multi-modal model trained with a recipe adapted from text-only language models, including a large-scale retrieval-augmented pre-training stage and a second multi-task supervised fine-tuning (SFT) stage. It is also a general-purpose model that can do both text-to-image and image-to-text generation, allowing us to introduce self-contained contrastive decoding methods that produce high-quality outputs. Extensive experiments demonstrate that this recipe is highly effective for multi-modal models. CM3Leon achieves state-of-the-art performance in text-to-image generation with 5x less training compute than comparable methods (zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate unprecedented levels of controllability in tasks ranging from language-guided image editing to image-controlled generation and segmentation.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 41 references that are in the map. First posted on arXiv 5 Sep 2023.

Builds on

37 of 41 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
25 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  13. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  14. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  15. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  16. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  17. 2022Understanding Diffusion Models: A Unified Perspectivecited by 337
  18. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  19. 2019Connecting Vision and Language with Localized Narrativescited by 225
  20. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 208
  21. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  22. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  23. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  24. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  25. 2022Faster Maximum Inner Product Search in High Dimensionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 39; 2024: 88; 2025: 27; 2026: 14.

168 papers.

Sort:

By area:

  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  2. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  3. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  4. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  5. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  6. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  7. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  8. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  9. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  10. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  11. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  12. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
38 more
  1. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  2. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  3. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  5. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  6. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  7. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  8. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  9. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  10. 2024An Introduction to Vision-Language Modelingcited by 175
  11. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 174
  12. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  13. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  14. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  15. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  16. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  17. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  18. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  19. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  20. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  21. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 112
  22. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  23. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
  24. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  25. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  26. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  27. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  28. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  29. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  30. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 70
  31. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 69
  32. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 62
  33. 2023ControlRoom3D: Room Generation using Semantic Proxy Roomscited by 54
  34. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  35. 2023Generative Models: What Do They Know? Do They Know Things? Let's Find Out!cited by 50
  36. 2023Contrastive Denoising Score for Text-guided Latent Diffusion Image Editingcited by 49
  37. 2023SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformerscited by 49
  38. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49

50 most cited of 168 All 168 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval
  2. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
  3. 2025HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding
  4. 2024MIO: A Foundation Model on Multimodal Tokens
  5. 2023Generating Images with Multimodal Language Models
  6. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
  7. 2024A Review of Multi-Modal Large Language and Vision Models
  8. 2023SwitchGPT: Adapting Large Language Models for Non-Text Outputs