mlsanityData to 7 Oct 2026

What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration

arXiv 2410.20482 ↗First posted 27 Oct 2024cs.CL cs.AI cs.CVcited by 31

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Recently, rapid advancements in Multi-Modal In-Context Learning (MM-ICL) have achieved notable success, which is capable of achieving superior performance across various tasks without requiring additional parameter tuning. However, the underlying rules for the effectiveness of MM-ICL remain under-explored. To fill this gap, this work aims to investigate the research question: "What factors affect the performance of MM-ICL?'' To this end, we investigate extensive experiments on the three core steps of MM-ICL including demonstration retrieval, demonstration ordering, and prompt construction using 6 vision large language models and 20 strategies. Our findings highlight (1) the necessity of a multi-modal retriever for demonstration retrieval, (2) the importance of intra-demonstration ordering over inter-demonstration ordering, and (3) the enhancement of task comprehension through introductory instructions in prompts. We hope this study can serve as a foundational guide for optimizing MM-ICL strategies in future research.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 79 of its 89 references that are in the map. First posted on arXiv 27 Oct 2024.

Builds on

79 of 89 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought26 May
  2. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition7 May
  3. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models25 Oct
  4. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  5. 2022Visual Programming: Compositional visual reasoning without training18 Nov
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 73

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
38 more
  1. 2019Towards VQA Models That Can Readcited by 1,696
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2023A Survey on Multimodal Large Language Modelscited by 898
  11. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  12. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  13. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  14. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  15. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  16. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  17. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  18. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  19. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  20. 2019Scene Text Visual Question Answeringcited by 327
  21. 2024What matters when building vision-language models?cited by 321
  22. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  23. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  24. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  25. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  26. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  27. 2023Looped Transformers as Programmable Computerscited by 224
  28. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  29. 2022In-context Examples Selection for Machine Translationcited by 172
  30. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  31. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  32. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  33. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  34. 2022End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Modelscited by 105
  35. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  36. 2023Finding Support Examples for In-Context Learningcited by 104
  37. 2024Large Language Models Meet NLP: A Surveycited by 102
  38. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 18; 2026: 11.

31 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  3. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  4. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 34
  5. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 29
  6. 2025ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Modelscited by 18
  7. 2025Analyzing Finetuning Representation Shift for Multimodal LLMs Steeringcited by 15
  8. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 14
  9. 2025X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic Systemcited by 10
  10. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 9
  11. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Modelscited by 7
  12. 2026Efficient and Minimax Optimal In-context Nonparametric Regression with Transformerscited by 7
19 more
  1. 2025Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoderscited by 6
  2. 2025Electronic Circuit Principles of Large Language Modelscited by 4
  3. 2025An Empirical Study on Configuring In-Context Learning Demonstrations for Unleashing MLLMs' Sentimental Perception Capabilitycited by 4
  4. 2025What do vision-language models see in the context? Investigating multimodal in-context learningcited by 4
  5. 2025MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderationcited by 3
  6. 2024Exploring Multi-Grained Concept Annotations for Multimodal Large Language Modelscited by 2
  7. 2025MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Modelscited by 2
  8. 2025AutoPR: Let's Automate Your Academic Promotion!cited by 2
  9. 2026UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomycited by 2
  10. 2026ArtifactLens: Hundreds of Labels Are Enough for Artifact Detection with VLMscited by 1
  11. 2026Large Multimodal Models as General In-Context Classifierscited by 1
  12. 2026HIFICL: High-Fidelity In-Context Learning for Multimodal Taskscited by 1
  13. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 1
  14. 2025Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMscited by 0
  15. 2026Parallel In-context Learning for Large Vision Language Modelscited by 0
  16. 2026Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learningcited by 0
  17. 2026Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Taskcited by 0
  18. 2026When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICLcited by 0
  19. 2026QwenVLConnector: A Fast, Unified Medical VLM Chatbot for Fine-Grained Clinical Perception and Text Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
  2. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning
  3. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  4. 2025Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning
  5. 2025On the Relationship Between the Choice of Representation and In-Context Learning
  6. 2023Dr.ICL: Demonstration-Retrieved In-context Learning
  7. 2023MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examples
  8. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations