mlsanityData to 7 Oct 2026

Otter: A Multi-Modal Model with In-Context Instruction Tuning

arXiv 2305.03726 ↗First posted 5 May 2023cs.CV cs.CLcited by 548

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent advances in Large Multimodal Models (LMMs) have unveiled great potential as visual assistants. However, most existing works focus on responding to individual instructions or using previous dialogues for contextual understanding. There is little discussion on employing both images and text as in-context examples to enhance the instruction following capability. To bridge this gap, we introduce the Otter model to leverage both textual and visual in-context examples for instruction tuning. Specifically, Otter builds upon Flamingo with Perceiver architecture, and has been instruction tuned for general purpose multi-modal assistant. Otter seamlessly processes multi-modal inputs, supporting modalities including text, multiple images, and dynamic video content. To support the training of Otter, we present the MIMIC-IT (MultI-Modal In-Context Instruction Tuning) dataset, which encompasses over 3 million multi-modal instruction-response pairs, including approximately 2.2 million unique instructions across a broad spectrum of images and videos. MIMIC-IT has been carefully curated to feature a diverse array of in-context examples for each entry. Comprehensive evaluations suggest that instruction tuning with these in-context examples substantially enhances model convergence and generalization capabilities. Notably, the extensive scenario coverage provided by the MIMIC-IT dataset empowers the Otter model to excel in tasks involving complex video and multi-image understanding.

Key facts. Cited by 548 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 109 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

58 of 109 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2019Towards VQA Models That Can Readcited by 1,696
  13. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  14. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  15. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  16. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  17. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  18. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  19. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  20. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  21. 2016Modeling Context in Referring Expressionscited by 1,054
  22. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  23. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  24. 2017Dense-Captioning Events in Videoscited by 789
  25. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  26. 2022Transformers learn in-context by gradient descentcited by 714
  27. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  28. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  29. 2023Instruction Tuning with GPT-4cited by 676
  30. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  31. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  32. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  33. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  34. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  35. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  36. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  37. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  38. 2015Visual7W: Grounded Question Answering in Imagescited by 315

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 182; 2024: 248; 2025: 81; 2026: 37.

548 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
38 more
  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  2. 2023A Survey on Multimodal Large Language Modelscited by 898
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  4. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  7. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  8. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  9. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  11. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  12. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  13. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  14. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  15. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  16. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  17. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  18. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  19. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  20. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  21. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  22. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  23. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  24. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  25. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  26. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  27. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 320
  28. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  29. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  30. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  31. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  32. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  33. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  34. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  35. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  36. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  37. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  38. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271

50 most cited of 548 All 548 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuning
  2. 2023Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?
  3. 2025CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMs
  4. 2023On the Performance of Multimodal Language Models
  5. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning
  6. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
  7. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  8. 2025MM-IFEngine: Towards Multimodal Instruction Following