mlsanityData to 7 Oct 2026

What Makes Multimodal In-Context Learning Work?

arXiv 2404.15736 ↗First posted 24 Apr 2024cs.CV cs.AIcited by 54

Not on the map. Why?

Cited by 37 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models have demonstrated remarkable performance across various tasks, exhibiting the capacity to swiftly acquire new skills, such as through In-Context Learning (ICL) with minimal demonstration examples. In this work, we present a comprehensive framework for investigating Multimodal ICL (M-ICL) in the context of Large Multimodal Models. We consider the best open-source multimodal models (e.g., IDEFICS, OpenFlamingo) and a wide range of multimodal tasks. Our study unveils several noteworthy findings: (1) M-ICL primarily relies on text-driven mechanisms, showing little to no influence from the image modality. (2) When used with advanced-ICL strategy (like RICES), M-ICL is not better than a simple strategy based on majority voting over context examples. Moreover, we identify several biases and limitations of M-ICL that warrant consideration prior to deployment. Code available at https://gitlab.com/folbaeni/multimodal-icl

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 58 of its 72 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

58 of 72 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
38 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2022Transformers learn in-context by gradient descentcited by 714
  12. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  13. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  14. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  15. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  16. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  17. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  18. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  19. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  20. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  21. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  22. 2023Larger language models do in-context learning differentlycited by 386
  23. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  24. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  25. 2023In-Context Learning Creates Task Vectorscited by 302
  26. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  27. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  28. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  29. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  30. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  31. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  32. 20234M: Massively Multimodal Masked Modelingcited by 118
  33. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  34. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  35. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  36. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  37. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  38. 2023Symbol tuning improves in-context learning in language modelscited by 72

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 22; 2026: 22.

54 papers.

Sort:

By area:

  1. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 48
  2. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  3. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  4. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 24
  5. 2025CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learningcited by 18
  6. 2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
  7. 2025Analyzing Finetuning Representation Shift for Multimodal LLMs Steeringcited by 15
  8. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  9. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 14
  10. 2024From Efficient Multimodal Models to World Models: A Surveycited by 13
  11. 2025Mimic In-Context Learning for Multimodal Taskscited by 12
  12. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimizationcited by 10
38 more
  1. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  2. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 9
  3. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Modelscited by 7
  4. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 6
  5. 2025SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learningcited by 5
  6. 2025Exposing and Mitigating Calibration Biases and Demographic Unfairness in MLLM Few-Shot In-Context Learning for Medical Image Classificationcited by 5
  7. 2026Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformerscited by 5
  8. 2024Adapting Large Multimodal Models to Distribution Shifts: The Role of In-Context Learningcited by 4
  9. 2025An Empirical Study on Configuring In-Context Learning Demonstrations for Unleashing MLLMs' Sentimental Perception Capabilitycited by 4
  10. 2025What do vision-language models see in the context? Investigating multimodal in-context learningcited by 4
  11. 2024Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?cited by 3
  12. 2025A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospectscited by 2
  13. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysiscited by 2
  14. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  15. 2025Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Modelscited by 2
  16. 2025Online In-Context Distillation for Low-Resource Vision Language Modelscited by 2
  17. 2026VIOLA: Towards Video In-Context Learning with Minimal Annotationscited by 2
  18. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMscited by 2
  19. 2025Classifying the Unknown: In-Context Learning for Open-Vocabulary Text and Symbol Recognitioncited by 1
  20. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 1
  21. 2026FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliancecited by 1
  22. 2026Large Multimodal Models as General In-Context Classifierscited by 1
  23. 2026Non-Parametric Structural Priors for Geometry Theorem Predictioncited by 1
  24. 2026Few-shot Writer Adaptation via Multimodal In-Context Learningcited by 1
  25. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 1
  26. 2026How to Ask the AI: A User Perspective Survey for Large Language Model Promptingcited by 1
  27. 2024Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learningcited by 0
  28. 2025Exploring Imbalanced Annotations for Effective In-Context Learningcited by 0
  29. 2025Vision-Language In-Context Learning Driven Few-Shot Visual Inspection Modelcited by 0
  30. 2025Markov Chain Estimation with In-Context Learningcited by 0
  31. 2026Ego: Embedding-Guided Personalization of Vision-Language Modelscited by 0
  32. 2026Parallel In-context Learning for Large Vision Language Modelscited by 0
  33. 2026Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learningcited by 0
  34. 2026Decompose, Compare, and Decide: Multimodal LLMs are Implicit Few-Shot Learnerscited by 0
  35. 2026Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Taskcited by 0
  36. 2026Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Modelscited by 0
  37. 2026Global Logic and Local Search: Dual-Stream Multimodal In-Context Learning for Verifiable Industrial Anomaly Detectioncited by 0
  38. 2026Gradient-free Task-Conditioned Retrieval for On-Device In-Context Learningcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
  2. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
  3. 2024Can MLLMs Perform Text-to-Image In-Context Learning?
  4. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning
  5. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  6. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations
  7. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  8. 2024How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?