mlsanityData to 14 Oct 2026

Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

arXiv 2604.13403 ↗First posted 15 Apr 2026cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. Despite its success in large language models, the extension of ICL to multimodal settings remains poorly understood in terms of its internal mechanisms and how it differs from text-only ICL. In this work, we conduct a systematic analysis of ICL in multimodal large language models. Using identical task formulations across modalities, we show that multimodal ICL performs comparably to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations. To understand this gap, we decompose multimodal ICL into task mapping construction and task mapping transfer, and analyze how models establish cross-modal task mappings, and transfer them to query samples across layers. Our analysis reveals that current models lack reasoning-level alignment between visual and textual representations, and fail to reliably transfer learned task mappings to queries. Guided by these findings, we further propose a simple inference-stage enhancement method that reinforces task mapping transfer. Our results provide new insights into the mechanisms and limitations of multimodal ICL and suggest directions for more effective multimodal adaptation. Our code is available hrefhttps://github.com/deeplearning-wisc/Multimocal-ICL-Analysis-Framework-MGIhere.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

32 of 45 references.

References · 32

  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  5. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  6. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  7. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  8. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  9. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  10. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  11. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  12. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 89
20 more
  1. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
  2. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  3. 2024What Makes Multimodal In-Context Learning Work?cited by 56
  4. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 55
  5. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 51
  6. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  7. 2023Understanding In-Context Learning via Supportive Pretraining Datacited by 41
  8. 2024An Information-Theoretic Analysis of In-Context Learningcited by 40
  9. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  10. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  11. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 34
  12. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  13. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configurationcited by 23
  14. 2025CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learningcited by 18
  15. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 16
  16. 2024How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?cited by 13
  17. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learningcited by 11
  18. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrationscited by 7
  19. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learningcited by 4
  20. 2025An Empirical Study on Configuring In-Context Learning Demonstrations for Unleashing MLLMs' Sentimental Perception Capabilitycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactualscited by 0
  2. 2026In-Context Learning Can Help Vision-Language Models Overcome Training Priorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  2. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  3. 2026Beyond Surface Imitation: Contrastive Modeling for Reasoning Path Alignment in Multimodal In-Context Learning
  4. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
  5. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  6. 2026UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy
  7. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  8. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning