mlsanityData to 14 Oct 2026

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

arXiv 2610.12942 ↗First posted 9 Oct 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning also fits: Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to properly respond to visual evidence when input images violate those regularities. It is challenging to disambiguate these prior-induced errors from errors caused by the model missing the underlying visual capabilities. In this work, we use controlled visual in-context learning to encourage the model to focus on the visual evidence, and demonstrate that this alone is frequently sufficient to correct the errors -- suggesting that the models in fact do possess the relevant visual capabilities. Specifically, we show that across five vision-centric tasks, demonstrations yield large gains on prior-conflicting counterfactual examples (+27.2% for Qwen3-VL 235B and +15.1% for Qwen3-VL 32B) while leaving accuracy on prior-aligned examples nearly unchanged. These findings generalize across five VLM families with gains from +9.7% to +30.3%. We analyze visual representations and find that demonstrations indeed redirect attention toward grounding cues while reducing reliance on cues that support the learned prior. Finally, we find that in-context learning is limited when the underlying visual capability is weak or absent. Together, these results show that current VLMs possess visual grounding capabilities that standard evaluations may overlook, underscoring the need for controlled evaluation design in diagnosing and improving model capability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 70 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

59 of 70 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Vision Language Models are Confused Tourists21 Nov
  2. 2025[De|Re]constructing VLMs' Reasoning in Counting22 Oct
  3. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting6 Oct
  4. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models9 Jun
  5. 2025Vision Language Models are Biased29 May
  6. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts21 May
  7. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding8 Mar
  8. 2025The in-context inductive biases of vision-language models differ across modalities3 Feb
  9. 2024Probing Visual Language Priors in VLMs31 Dec
  10. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Models13 Jun
  11. 2024IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models23 Mar
  12. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?21 Mar
2 more
  1. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training4 Mar
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2025Qwen3-VL Technical Reportcited by 2,793
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  10. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  11. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  12. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
33 more
  1. 2023Larger language models do in-context learning differentlycited by 394
  2. 2026Reinforcement Learning via Self-Distillationcited by 332
  3. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  4. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 224
  5. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  6. 2025ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemscited by 190
  7. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  8. 2022Learning by Distilling Contextcited by 138
  9. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  10. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 103
  11. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 75
  12. 2026MIRAGE: The Illusion of Visual Understandingcited by 65
  13. 2024What Makes Multimodal In-Context Learning Work?cited by 56
  14. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 51
  15. 2026BabyVision: Visual Reasoning Beyond Languagecited by 49
  16. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
  17. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 36
  18. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 34
  19. 2025In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learningcited by 22
  20. 2025Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learningcited by 19
  21. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 16
  22. 2025Visual Representations inside the Language Modelcited by 15
  23. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 15
  24. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learningcited by 11
  25. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 9
  26. 2024Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuningcited by 8
  27. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrationscited by 7
  28. 2023The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Modelscited by 6
  29. 2025Format as a Prior: Quantifying and Analyzing Bias in LLMs for Heterogeneous Datacited by 4
  30. 2026Recursive Belief Vision Language Action Modelscited by 4
  31. 2025VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Modelscited by 3
  32. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 2
  33. 2024ViSTa Dataset: Do vision-language models understand sequential tasks?cited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026In-Context Collapse in Vision-Language Models and How to Mitigate it?
  2. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  3. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
  4. 2024Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?
  5. 2024Visual In-Context Learning for Large Vision-Language Models
  6. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  7. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy
  8. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning