mlsanityData to 7 Oct 2026

Generative Multimodal Models are In-Context Learners

arXiv 2312.13286 ↗First posted 20 Dec 2023cs.CVcited by 446

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstrate that the task-agnostic in-context learning capabilities of large multimodal models can be significantly enhanced by effective scaling-up. We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences with a unified autoregressive objective. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning, such as visual prompting and object-grounded generation. The model sets a new record on multiple multimodal understanding tasks in few-shot settings. When instruction-tuned to follow specific instructions, Emu2 further achieves new state-of-the-art on challenging tasks such as question answering benchmarks for large multimodal models and open-ended subject-driven generation. These achievements demonstrate that Emu2 can serve as a base model and general-purpose interface for a wide range of multimodal tasks. Code and models are publicly available to facilitate future research.

Key facts. Cited by 446 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 80 of its 94 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

80 of 94 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Models4 Oct
  3. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  4. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  5. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
  6. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models25 Aug
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  10. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  11. 2023Generating Images with Multimodal Language Models26 May
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
2 more
  1. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  2. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  13. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  14. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  15. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  16. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  17. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  18. 2019Towards VQA Models That Can Readcited by 1,696
  19. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  20. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  21. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  22. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  23. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  24. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  25. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  26. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  27. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  28. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  29. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  30. 2023ModelScope Text-to-Video Technical Reportcited by 651
  31. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  32. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  33. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  34. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  35. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  36. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  37. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  38. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 175; 2025: 182; 2026: 88.

446 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  3. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  4. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  5. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  6. 2024OmniGen: Unified Image Generationcited by 388
  7. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  8. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  9. 2024What matters when building vision-language models?cited by 321
  10. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  11. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  12. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
38 more
  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  2. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  3. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  4. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  5. 2025Transfer between Modalities with MetaQueriescited by 231
  6. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  7. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  8. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  9. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  10. 2023Video Understanding with Large Language Models: A Surveycited by 203
  11. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  12. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  13. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  14. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  15. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  16. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  17. 2024In-Context LoRA for Diffusion Transformerscited by 155
  18. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  19. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 142
  20. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  21. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  22. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  23. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  24. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  25. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  26. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  27. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  28. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 112
  29. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  30. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  31. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  32. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  33. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 100
  34. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  35. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
  36. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  37. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  38. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90

50 most cited of 446 All 446 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning
  2. 2023On the Performance of Multimodal Language Models
  3. 2023Emu: Generative Pretraining in Multimodality
  4. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering
  5. 2026Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards
  6. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
  7. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings
  8. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks