mlsanityData to 14 Oct 2026

Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learning

arXiv 2608.23570 ↗First posted 14 Jun 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large vision-language models exhibit strong in-context learning (ICL) capabilities, yet when and why visual context helps multimodal ICL remains poorly understood. Empirical studies show a puzzling dichotomy: models sometimes effectively leverage visual demonstrations, yet often neglect them entirely. We propose VIB-ICL, an information-theoretic framework that resolves this dichotomy through the Information Bottleneck principle. We introduce the Cross-Modal Information Gain (CMIG), which quantifies the additional mutual information that visual context provides about the target beyond textual context. We derive a generalization bound showing that multimodal ICL's excess risk over text-only ICL is governed by the CMIG, proving that multimodal ICL provably outperforms text-only ICL when visual information is non-redundant. We further prove that visual context neglect, often viewed as a failure mode, is the Information Bottleneck-optimal solution when visual information is redundant, yielding a closed-form Attention Reallocation Principle that prescribes how visual attention weights should be adaptively adjusted. We instantiate this principle in the VIB-ICL algorithm, which estimates CMIG via variational bounds and dynamically reallocates attention. Experiments on five benchmarks demonstrate consistent improvements of up to 4.7% accuracy gains and 35% reduction in required demonstrations, validating our theoretical predictions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 37 references that are in the map. First posted on arXiv 14 Jun 2026.

Builds on

33 of 37 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025What do vision-language models see in the context? Investigating multimodal in-context learning28 Oct
  2. 2025ContextNav: Towards Agentic Multimodal In-Context Learning6 Oct
  3. 2025True Multimodal In-Context Learning Needs Attention to the Visual Context21 Jul
  4. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimization19 Apr
  5. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering6 Apr
  6. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning21 Jun
  7. 2024Towards Multimodal In-Context Learning for Vision & Language Models19 Mar
  8. 2023Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning5 Dec
  9. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep

Other references · 24

  1. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  2. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 128
  3. 2023SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agentscited by 71
  4. 2024What Makes Multimodal In-Context Learning Work?cited by 56
  5. 2025Improving Medical Large Vision-Language Models with Abnormal-Aware Feedbackcited by 42
  6. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 36
  7. 2024GATEAU: Selecting Influential Samples for Long Context Alignmentcited by 20
  8. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  9. 2025DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignmentcited by 16
  10. 2024X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Modelscited by 15
  11. 2025Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filteringcited by 12
  12. 2025DSPC: Dual-Stage Progressive Compression Framework for Efficient Long-Context Reasoningcited by 10
12 more
  1. 2024Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lenscited by 8
  2. 2024In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learningcited by 6
  3. 2024Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learningcited by 6
  4. 2025Advancing Autonomous VLM Agents via Variational Subgoal-Conditioned Reinforcement Learningcited by 5
  5. 2025U-ViLAR: Uncertainty-Aware Visual Localization for Autonomous Driving via Differentiable Association and Registrationcited by 5
  6. 2024In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Modelcited by 4
  7. 2024Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?cited by 3
  8. 2024In-context learning for medical image segmentationcited by 3
  9. 2024In-Context Learning of Energy Functionscited by 2
  10. 2025Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learningcited by 2
  11. 2025In-Context Learning for Seismic Data Processingcited by 1
  12. 2025HGAN-SDEs: Learning Neural Stochastic Differential Equations with Hermite-Guided Adversarial Trainingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
  2. 2026In-Context Learning Can Help Vision-Language Models Overcome Training Priors
  3. 2025Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning
  4. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  5. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
  6. 2026CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models
  7. 2025Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning
  8. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models