mlsanityData to 7 Oct 2026

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

arXiv 2309.07915 ↗First posted 14 Sep 2023cs.CL cs.AI cs.CVcited by 171

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Since the resurgence of deep learning, vision-language models (VLMs) enhanced by large language models (LLMs) have grown exponentially in popularity. However, while LLMs can utilize extensive background knowledge and task information with in-context learning, most VLMs still struggle with understanding complex multi-modal prompts with multiple images, making VLMs less effective in downstream vision-language tasks. In this paper, we address the limitation above by 1) introducing vision-language Model with Multi-Modal In-Context Learning(MMICL), a new approach to allow the VLM to deal with multi-modal inputs efficiently; 2) proposing a novel context scheme to augment the in-context learning ability of the VLM; 3) constructing the Multi-modal In-Context Learning (MIC) dataset, designed to enhance the VLM's ability to understand complex multi-modal prompts. Our experiments confirm that MMICL achieves new state-of-the-art zero-shot performance on a wide range of general vision-language tasks, especially for complex benchmarks, including MME and MMBench. Our analysis demonstrates that MMICL effectively tackles the challenge of complex multi-modal prompt understanding and emerges the impressive ICL ability. Furthermore, we observe that MMICL successfully alleviates language bias in VLMs, a common issue for VLMs that often leads to hallucination when faced with extensive textual context. Our code, dataset, dataset tool, and model are available at https://github.com/PKUnlp-icler/MIC

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 83 of its 140 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

83 of 140 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  5. 2023What You See is What You Read? Improving Text-Image Alignment Evaluation17 May
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  7. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  8. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 75

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  14. 2016Modeling Context in Referring Expressionscited by 1,054
  15. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  16. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  17. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  18. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  19. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  20. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  21. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  22. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  23. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  24. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  25. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  26. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  27. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  28. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  29. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  30. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  31. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  32. 2021MetaICL: Learning to Learn In Contextcited by 450
  33. 2018nocaps: novel object captioning at scalecited by 444
  34. 2016Visual Dialogcited by 377
  35. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  36. 2019Scene Text Visual Question Answeringcited by 327
  37. 2022Visual Spatial Reasoningcited by 288
  38. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 27; 2024: 82; 2025: 43; 2026: 19.

171 papers.

Sort:

By area:

  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  3. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  4. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  5. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  6. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  7. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  8. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  9. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  10. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  11. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  12. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
38 more
  1. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  2. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  3. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  4. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  5. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  6. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  7. 2023Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Modelscited by 104
  8. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  9. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  10. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 74
  11. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  12. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  13. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automationcited by 70
  14. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 64
  15. 2024Red Teaming Visual Language Modelscited by 62
  16. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputscited by 56
  17. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  18. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  19. 2023Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizercited by 53
  20. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 50
  21. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  22. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  23. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  24. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  25. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 43
  26. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 42
  27. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  28. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 34
  29. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  30. 2023The Role of Chain-of-Thought in Complex Vision-Language Reasoning Taskcited by 32
  31. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  32. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 31
  33. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 31
  34. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  35. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learningcited by 30
  36. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agentscited by 30
  37. 2023Prompt Highlighter: Interactive Control for Multi-Modal LLMscited by 29
  38. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chaincited by 29

50 most cited of 171 All 171 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  2. 2025MMRL: Multi-Modal Representation Learning for Vision-Language Models
  3. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  4. 2025MMRL++: Parameter-Efficient and Interaction-Aware Representation Learning for Vision-Language Models
  5. 2023MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  7. 2023MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examples
  8. 2026MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation