mlsanityData to 7 Oct 2026

From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning

arXiv 2407.00902 ↗First posted 1 Jul 2024cs.CV cs.AI cs.CLcited by 10

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Motivated by in-context learning (ICL) capabilities of Large Language Models (LLMs), multimodal LLMs with additional visual modality are also exhibited with similar ICL abilities when multiple image-text pairs are provided as demonstrations. However, relatively less work has been done to investigate the principles behind how and why multimodal ICL works. We conduct a systematic and principled evaluation of multimodal ICL for models of different scales on a broad spectrum of new yet critical tasks. Through perturbations over different modality information, we show that modalities matter differently across tasks in multimodal ICL. Guided by task-specific modality impact, we recommend modality-driven demonstration strategies to boost ICL performance. We also find that models may follow inductive biases from multimodal ICL even if they are rarely seen in or contradict semantic priors from pretraining data. Our principled analysis provides a comprehensive way of understanding the role of demonstrations in multimodal in-context learning, and sheds light on effectively improving multimodal ICL on a wide range of tasks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 62 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

41 of 62 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension25 Apr
  2. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  3. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models5 Dec
  4. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
25 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  7. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  10. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  11. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  12. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  13. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  14. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  15. 2023Larger language models do in-context learning differentlycited by 386
  16. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  17. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  18. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  19. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  20. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  21. 2020PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphonescited by 116
  22. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  23. 2023Coverage-based Example Selection for In-Context Learningcited by 58
  24. 2024Large Language Models Lack Understanding of Character Composition of Wordscited by 21
  25. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 2.

10 papers.

  1. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Modelscited by 28
  2. 2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
  3. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 14
  4. 2025Mimic In-Context Learning for Multimodal Taskscited by 12
  5. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Modelscited by 7
  6. 2025Retrieval-augmented in-context learning for multimodal large language models in disease classificationcited by 4
  7. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysiscited by 2
  8. 2025Classifying the Unknown: In-Context Learning for Open-Vocabulary Text and Symbol Recognitioncited by 1
  9. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 1
  10. 2026MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  2. 2024What Makes Multimodal In-Context Learning Work?
  3. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning
  4. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
  5. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration
  6. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  7. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  8. 2025On the Relationship Between the Choice of Representation and In-Context Learning