mlsanityData to 7 Oct 2026

Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning

arXiv 2511.08246 ↗First posted 11 Nov 2025cs.AIcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Multimodal Models (LMMs) have shown promising in-context learning (ICL) capabilities, but scaling to many-shot settings remains difficult due to limited context length and high inference cost. To address these challenges, task-vector-based methods have been explored by inserting compact representations of many-shot in-context demonstrations into model activations. However, existing task-vector-based methods either overlook the importance of where to insert task vectors or struggle to determine suitable values for each location. To this end, we propose a novel Sensitivity-aware Task Vector insertion framework (STV) to figure out where and what to insert. Our key insight is that activation deltas across query-context pairs exhibit consistent structural patterns, providing a reliable cue for insertion. Based on the identified sensitive-aware locations, we construct a pre-clustered activation bank for each location by clustering the activation values, and then apply reinforcement learning to choose the most suitable one to insert. We evaluate STV across a range of multimodal models (e.g., Qwen-VL, Idefics-2) and tasks (e.g., VizWiz, OK-VQA), demonstrating its effectiveness and showing consistent improvements over previous task-vector-based methods with strong generalization.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 43 references that are in the map. First posted on arXiv 11 Nov 2025.

Builds on

26 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Mimic In-Context Learning for Multimodal Tasks11 Apr
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning21 Jun
  4. 2024On the Out-Of-Distribution Generalization of Multimodal Large Language Models9 Feb
  5. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  10. 2023In-Context Learning Creates Task Vectorscited by 302
  11. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  12. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
9 more
  1. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  2. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 64
  3. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  4. 2023In-Context Learning with Many Demonstration Examplescited by 39
  5. 2023How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigationcited by 22
  6. 2019apricot: Submodular selection for data summarization in Pythoncited by 21
  7. 2025Task Vectors in In-Context Learning: Emergence, Formation, and Benefitcited by 20
  8. 2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
  9. 2025Efficient Multi-modal Long Context Learning for Training-free Adaptationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026CoEvolve: Training LLM Agents via Agent-Data Mutual Evolutioncited by 14
  2. 2026PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Mergingcited by 2
  3. 2026AR-MAP: Are Autoregressive Large Language Models Implicit Teachers for Diffusion Large Language Models?cited by 1
  4. 2026Towards High-Resolution Visual Perception via Hierarchical Entity Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks
  2. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations
  3. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
  4. 2026UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy
  5. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration
  6. 2026HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks
  7. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering
  8. 2024Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback