mlsanityData to 14 Oct 2026

In-Context Collapse in Vision-Language Models and How to Mitigate it?

arXiv 2608.02830 ↗First posted 3 Aug 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image--label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied. We show the opposite: as demonstrations accumulate, a subset of VLMs undergo an in-context collapse, a sharp, sometimes catastrophic accuracy drop spanning synthetic classification, natural-image classification, and VQA benchmarks, in some models falling below chance while outputs remain well-formed. Across an open VLM panel (0.5B--11B) and a frontier model (Claude Sonnet 4.5), the collapse is graded. Two capabilities turn out to be dissociable: robustness to accumulating demonstrations and the ability to learn a novel rule in context, their combinations yield three reproducible regimes. A parameter-matched lesion-and-rescue causally localizes the collapse to the vision-language integration pathway: an adapter on the connector and early/mid layers restores genuine learning (remap accuracy 0.39!rightarrow!0.91 at 16 shots), while an equal-capacity adapter on the late readout does not. We propose textscCircA, whose core is a one-time integration vaccine: trained once on one synthetic task, it transfers collapse-resistance to unseen task families (chancerightarrow0.71/0.60 on CIFAR/Fashion). The layers best for in-context integration are not the layers best for weight-based consolidation, the late readout achieves higher accuracy and less forgetting at fewer parameters. The collapse is an integration failure at the vision--language interface, correctable by a lightweight, transferable intervention.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 57 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

37 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models26 Nov

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  9. 2016Learning without Forgettingcited by 2,496
  10. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  12. 2017Continual Learning with Deep Generative Replaycited by 1,158
24 more
  1. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  2. 2022In-context Learning and Induction Headscited by 964
  3. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  6. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  7. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  8. 2023In-Context Learning Creates Task Vectorscited by 312
  9. 2023Function Vectors in Large Language Modelscited by 308
  10. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  11. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  12. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  13. 2022CLiMB: A Continual Learning Benchmark for Vision-and-Language Taskscited by 68
  14. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 67
  15. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  16. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  17. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 36
  18. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Promptcited by 30
  19. 2023Link-Context Learning for Multimodal LLMscited by 24
  20. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuningcited by 19
  21. 2025Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learningcited by 19
  22. 2021Cognitively Inspired Learning of Incremental Drifting Conceptscited by 16
  23. 2024Dynamic Transformer Architecture for Continual Learning of Multimodal Taskscited by 8
  24. 2025Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Modelscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026In-Context Learning Can Help Vision-Language Models Overcome Training Priors
  2. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  3. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  4. 2024Visual In-Context Learning for Large Vision-Language Models
  5. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  6. 2024Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?
  7. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
  8. 2026TraceICL: Thread-Resolved Demonstration Curation with Consistency Calibration for Robust Multimodal In-Context Learning