mlsanityData to 7 Oct 2026

Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?

arXiv 2311.18021 ↗First posted 29 Nov 2023cs.CV cs.AI cs.LGcited by 31

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Language Models (LLMs) with in-context learning (ICL) ability can quickly adapt to a specific context given a few demonstrations (demos). Recently, Multimodal Large Language Models (MLLMs) built upon LLMs have also shown multimodal ICL ability, i.e., responding to queries given a few multimodal demos, including images, queries, and answers. While ICL has been extensively studied on LLMs, its research on MLLMs remains limited. One essential question is whether these MLLMs can truly conduct multimodal ICL, or if only the textual modality is necessary. We investigate this question by examining two primary factors that influence ICL: 1) Demo content, i.e., understanding the influences of demo content in different modalities. 2) Demo selection strategy, i.e., how to select better multimodal demos for improved performance. Experiments revealed that multimodal ICL is predominantly driven by the textual content whereas the visual information in the demos has little influence. Interestingly, visual content is still necessary and useful for selecting demos to increase performance. Motivated by our analysis, we propose a simple yet effective approach, termed Mixed Modality In-Context Example Selection (MMICES), which considers both visual and language modalities when selecting demos. Extensive experiments are conducted to support our findings and verify the improvement brought by our method. Code is available at urlhttps://chenxshuo.github.io/m-icl/.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 60 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

45 of 60 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  2. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  3. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2015VQA: Visual Question Answeringcited by 2,489
30 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2022A Survey on In-context Learningcited by 717
  11. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  12. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  13. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  14. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  15. 2023Larger language models do in-context learning differentlycited by 386
  16. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  17. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  18. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  19. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  20. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  21. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  22. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  23. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  24. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  25. 2023Learning to Retrieve In-Context Examples for Large Language Modelscited by 60
  26. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  27. 2023RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learningcited by 24
  28. 2023MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Modelscited by 12
  29. 2023SINC: Self-Supervised In-Context Learning for Vision-Language Taskscited by 4
  30. 2023Lightweight In-Context Tuning for Multimodal Unified Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 17.

31 papers.

Sort:

By area:

  1. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  2. 2025CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learningcited by 18
  3. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 14
  4. 2024Can MLLMs Perform Text-to-Image In-Context Learning?cited by 13
  5. 2025Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Modelscited by 12
  6. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 9
  7. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Modelscited by 7
  8. 2025Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoderscited by 6
  9. 2025SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learningcited by 5
  10. 2026Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformerscited by 5
  11. 2025MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderationcited by 3
  12. 2026Retrieving Counterfactuals Improves Visual In-Context Learningcited by 3
19 more
  1. 2024Large Visual-Language Models Are Also Good Classifiers: A Study of In-Context Multimodal Fake News Detectioncited by 2
  2. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  3. 2025Online In-Context Distillation for Low-Resource Vision Language Modelscited by 2
  4. 2025From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Imagescited by 2
  5. 2026UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomycited by 2
  6. 2026HIFICL: High-Fidelity In-Context Learning for Multimodal Taskscited by 1
  7. 2026Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selectioncited by 1
  8. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 1
  9. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigationcited by 1
  10. 2026Parallel In-context Learning for Large Vision Language Modelscited by 0
  11. 2026GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Modelscited by 0
  12. 2026Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Modelscited by 0
  13. 2026In-Context Learning for Wound Classification with Small Multimodal Language Modelscited by 0
  14. 2026Gradient-free Task-Conditioned Retrieval for On-Device In-Context Learningcited by 0
  15. 2026MAG: MAnifold Guided Semi-Supervised Multi-modal In-Context Learningcited by 0
  16. 2026When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICLcited by 0
  17. 2026TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactualscited by 0
  18. 2026Are In-Context Images Worth 10 Dimensions?cited by 0
  19. 2026MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024What Makes Multimodal In-Context Learning Work?
  2. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning
  3. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
  4. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  5. 2024Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
  6. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  7. 2024How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?
  8. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis