mlsanityData to 14 Oct 2026

Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning

arXiv 2505.17097 ↗First posted 21 May 2025cs.CV cs.CLcited by 19

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal in-context learning (ICL) is becoming a key capability that allows large vision-language models (LVLMs) to adapt to novel tasks without parameter updates, which expands their usefulness in many real-world applications. However, ICL performance remains unstable even when the in-context demonstrations (ICDs) are well matched, showing that LVLMs still struggle to make full use of the provided context. While existing work mainly focuses on prompt engineering or post-hoc logit calibration, we study the attention mechanisms inside LVLMs to address their inherent limitations. We identify two important weaknesses in their self-attention that hinder effective ICL. To address these weaknesses, we propose Context-Aware Modulated Attention (CAMA), a training-free and plug-and-play method that dynamically adjusts attention logits based on the input in-context sequence. CAMA uses a two-stage modulation process that strengthens attention to semantically important tokens, especially visual ones. Across four LVLMs and seven benchmarks, CAMA consistently outperforms vanilla models and baselines, showing clear effectiveness and generalization. It can also activate the intended benefits of prompt engineering methods and remains robust across different sequence configurations. Therefore, CAMA opens up new directions for improving multimodal reasoning through a deeper understanding of attention dynamics.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 46 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

27 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May

Other references · 25

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2019Towards VQA Models That Can Readcited by 1,718
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  9. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  10. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  11. 2022A Survey on In-context Learningcited by 721
  12. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
13 more
  1. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  2. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  3. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  4. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  5. 2023The Learnability of In-Context Learningcited by 127
  6. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 125
  7. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 114
  8. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 89
  9. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 72
  10. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  11. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigationcited by 23
  12. 2024Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contextscited by 19
  13. 2023MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examplescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 13.

19 papers.

Sort:

By area:

  1. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  2. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 10
  3. 2026ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratchcited by 10
  4. 2025SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysiscited by 9
  5. 2025From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoningcited by 9
  6. 2025Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attentioncited by 8
  7. 2025Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarioscited by 8
  8. 2026Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitrationcited by 6
  9. 2026Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generationcited by 5
  10. 2026What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoningcited by 4
  11. 2026Personalize Your Large Vision-language Models With In-context Prompt Tuningcited by 3
  12. 2026A2TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clippingcited by 2
7 more
  1. 2026Semantic Granularity Navigation in Image Editingcited by 1
  2. 2026Think Bright, Diffuse Nice: Enhancing T2I-ICL via Inductive-Bias Hint Instruction and Query Contrastive Decodingcited by 0
  3. 2026Neutral-Reference Prompting for Vision-Language Modelscited by 0
  4. 2026Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Modelscited by 0
  5. 2026In-Context Collapse in Vision-Language Models and How to Mitigate it?cited by 0
  6. 2026MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Modelscited by 0
  7. 2026In-Context Learning Can Help Vision-Language Models Overcome Training Priorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025True Multimodal In-Context Learning Needs Attention to the Visual Context
  2. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  3. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  4. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations
  5. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering
  6. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  7. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
  8. 2024A-VL: Adaptive Attention for Large Vision-Language Models