mlsanityData to 14 Oct 2026

True Multimodal In-Context Learning Needs Attention to the Visual Context

arXiv 2507.15807 ↗First posted 21 Jul 2025cs.CV cs.AIcited by 16

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs), built on powerful language backbones, have enabled Multimodal In-Context Learning (MICL)-adapting to new tasks from a few multimodal demonstrations consisting of images, questions, and answers. Despite showing noticeable improvement on standard vision-language datasets, current MLLMs struggle to leverage visual information in the demonstrations. Specifically, they tend to neglect visual cues and over-rely on textual patterns, leading to mere text imitation rather than genuine multimodal adaptation. This behavior makes MICL still unimodal and largely restricts its practical utility. More importantly, this limitation is often concealed by the improved performance on tasks that do not require understanding the visual context. As a result, how to effectively enhance MICL ability and reliably evaluate the MICL performance remains underexplored. To address these issues, we first introduce Dynamic Attention Reallocation (DARA), an efficient fine-tuning strategy that encourages models to attend to the visual context by rebalancing attention across visual and textual tokens. In addition, we present TrueMICL, an MICL-dedicated dataset with both support and test sets that explicitly requires the integration of multimodal information-particularly visual content-for correct task completion. Extensive experiments demonstrate the effectiveness of our holistic solution, showcasing substantial improvements in the true multimodal in-context learning capabilities. Code and datasets are available at https://chenxshuo.github.io/true-micl-colm .

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 47 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

42 of 47 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration21 May
  2. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs24 Feb
  3. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization17 Nov
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  5. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images21 Jul
  6. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning1 Jul
  7. 2024Towards Multimodal In-Context Learning for Vision & Language Models19 Mar
  8. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?29 Nov
  9. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  11. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  12. 2024What matters when building vision-language models?cited by 326
20 more
  1. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  2. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  3. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  4. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  5. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 67
  6. 2024A Survey on LoRA of Large Language Modelscited by 67
  7. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  8. 2023Learning to Retrieve In-Context Examples for Large Language Modelscited by 60
  9. 2024What Makes Multimodal In-Context Learning Work?cited by 56
  10. 2024In-context learning enables multimodal large language models to classify cancer pathology imagescited by 38
  11. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  12. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 33
  13. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 31
  14. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  15. 2024In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translationcited by 22
  16. 2024How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?cited by 13
  17. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
  18. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Modelscited by 8
  19. 2024Visual Question Decomposition on Multimodal Large Language Modelscited by 7
  20. 2024Advanced Multimodal Deep Learning Architecture for Image-Text Matchingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 15.

16 papers.

Sort:

By area:

  1. 2026Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMscited by 8
  2. 2026Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformerscited by 5
  3. 2025Online In-Context Distillation for Low-Resource Vision Language Modelscited by 2
  4. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 2
  5. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 2
  6. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoningcited by 2
  7. 2026HIFICL: High-Fidelity In-Context Learning for Multimodal Taskscited by 1
  8. 2026Few-shot Writer Adaptation via Multimodal In-Context Learningcited by 1
  9. 2026Ego: Embedding-Guided Personalization of Vision-Language Modelscited by 0
  10. 2026Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problemscited by 0
  11. 2026Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Modelscited by 0
  12. 2026Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learningcited by 0
4 more
  1. 2026TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactualscited by 0
  2. 2026MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Modelscited by 0
  3. 2026In-Context Learning Can Help Vision-Language Models Overcome Training Priorscited by 0
  4. 2026ActICL: Post-Training for Agentic Multimodal In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  2. 2025Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning
  3. 2026Test-Time Computing for Referring Multimodal Large Language Models
  4. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning
  5. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  6. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations
  7. 2025Zoomer: Adaptive Image Focus Optimization for Black-box MLLM
  8. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models