mlsanityData to 14 Oct 2026

MIMIC-IT: Multi-Modal In-Context Instruction Tuning

arXiv 2306.05425 ↗First posted 8 Jun 2023cs.CV cs.AI cs.CLcited by 267

Not on the map. Why?

Cited by 170 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

High-quality instructions and responses are essential for the zero-shot performance of large language models on interactive natural language tasks. For interactive vision-language tasks involving intricate visual scenes, a large quantity of diverse and creative instruction-response pairs should be imperative to tune vision-language models (VLMs). Nevertheless, the current availability of vision-language instruction-response pairs in terms of quantity, diversity, and creativity remains limited, posing challenges to the generalization of interactive VLMs. Here we present MultI-Modal In-Context Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal instruction-response pairs, with 2.2 million unique instructions derived from images and videos. Each pair is accompanied by multi-modal in-context information, forming conversational contexts aimed at empowering VLMs in perception, reasoning, and planning. The instruction-response collection process, dubbed as Syphus, is scaled using an automatic annotation pipeline that combines human expertise with GPT's capabilities. Using the MIMIC-IT dataset, we train a large VLM named Otter. Based on extensive evaluations conducted on vision-language benchmarks, it has been observed that Otter demonstrates remarkable proficiency in multi-modal perception, reasoning, and in-context learning. Human evaluation reveals it effectively aligns with the user's intentions. We release the MIMIC-IT dataset, instruction-response collection pipeline, benchmarks, and the Otter model.

Key facts. Cited by 267 papers in the map (data through 14 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

42 of 55 references.

References · 42

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
30 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  13. 2017Dense-Captioning Events in Videoscited by 793
  14. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  15. 2023Instruction Tuning with GPT-4cited by 686
  16. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  17. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  18. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  19. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  20. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  21. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  22. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  23. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  24. 2016Visual Storytellingcited by 176
  25. 2022Learning by Distilling Contextcited by 138
  26. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  27. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  28. 2022Task-aware Retrieval with Instructionscited by 107
  29. 2022LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Taggingcited by 29
  30. 2022Making Your First Choice: To Address Cold Start Problem in Vision Active Learningcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 61; 2024: 123; 2025: 66; 2026: 17.

267 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  5. 2023A Survey on Multimodal Large Language Modelscited by 909
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  7. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  8. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  9. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  10. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  12. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
38 more
  1. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  2. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  3. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  4. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  5. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  6. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  7. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  8. 2024What matters when building vision-language models?cited by 326
  9. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  10. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  11. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  12. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  13. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  14. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  15. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
  16. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  17. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  18. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  19. 2024An Introduction to Vision-Language Modelingcited by 178
  20. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  21. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  22. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  23. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 154
  24. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  25. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  26. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  27. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 125
  28. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  29. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 121
  30. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  31. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  32. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 100
  33. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
  34. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  35. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 94
  36. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 91
  37. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 87
  38. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learningcited by 87

50 most cited of 267 All 267 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
  2. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  3. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  4. 2025Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
  5. 2023On the Performance of Multimodal Language Models
  6. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  7. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
  8. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use