mlsanityData to 14 Oct 2026

M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering

arXiv 2504.04633 ↗First posted 6 Apr 2025cs.CV cs.AIcited by 35

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Multimodal in-context learning (ICL) equips Large Vision-language Models (LVLMs) with the ability to adapt to new tasks via multiple user-provided demonstrations, without requiring any model parameter updates. However, its effectiveness is constrained by the token-intensive nature of multimodal inputs and the complexity of cross-modal few-shot reasoning, which together hinder LVLMs from extracting useful patterns from demonstrations. To address these challenges, we propose M2IV, a novel representation engineering approach that replaces explicit token-level demonstrations with a set of learnable Multimodal In-context Vectors directly injected into the residual streams of LVLMs. By analyzing the distinct roles of multi-head attention (MHA) and multi-layer perceptrons (MLP) in the ICL process, we design a training strategy that enables M2IV to perform fine-grained semantic distillation and robust cross-modal representation learning. M2IV not only improves performance across diverse tasks and LVLMs but also significantly reduces token overhead, enabling graceful scaling to many-shot scenarios. To further enhance usability, we introduce VLibrary, a repository that stores trained M2IVs for flexible retrieval and injection. With VLibrary, users can steer pre-trained LVLMs in a customized manner that meets diverse requirements. Extensive experiments demonstrate that M2IV consistently outperforms vanilla ICL and prior representation engineering baselines, achieving an average accuracy gain of 3.74% with substantial improvements in overall efficiency.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 55 references that are in the map. First posted on arXiv 6 Apr 2025.

Builds on

44 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations5 Mar
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images21 Jul
  5. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun

Other references · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  12. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
27 more
  1. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  4. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  5. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  6. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  7. 2024What matters when building vision-language models?cited by 326
  8. 2023In-Context Learning Creates Task Vectorscited by 312
  9. 2023Function Vectors in Large Language Modelscited by 308
  10. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  11. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  12. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  13. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  14. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  15. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  16. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 120
  17. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  18. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 98
  19. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 66
  20. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  21. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 60
  22. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 33
  23. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  24. 2024Implicit In-context Learningcited by 11
  25. 2024MIND: Multimodal Shopping Intention Distillation from Large Vision-language Models for E-commerce Purchase Understandingcited by 10
  26. 2024When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Modelscited by 6
  27. 2024II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answeringcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 21.

35 papers.

Sort:

By area:

  1. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
  2. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 42
  3. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configurationcited by 23
  4. 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigationcited by 23
  5. 2026TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrievalcited by 15
  6. 2026Steering Vector Fields for Context-Aware Inference-Time Control in Large Language Modelscited by 13
  7. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Codingcited by 12
  8. 2025Hy-Facial: Hybrid Feature Extraction by Dimensionality Reduction Methods for Enhanced Facial Expression Classificationcited by 10
  9. 2025MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMscited by 9
  10. 2025Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attentioncited by 8
  11. 2025FakeSV-VLM: Taming VLM for Detecting Fake Short-Video News via Progressive Mixture-Of-Experts Adaptercited by 7
  12. 2026Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitrationcited by 6
23 more
  1. 2026ActorMind: Emulating Human Actor Reasoning for Speech Role-Playingcited by 4
  2. 2025U2UData+: A Scalable Swarm UAVs Autonomous Flight Dataset for Embodied Long-horizon Taskscited by 3
  3. 2026HotComment: A Benchmark for Evaluating Popularity of Online Commentscited by 3
  4. 2026MM-StanceDet: Retrieval-Augmented Multi-modal Multi-agent Stance Detectioncited by 3
  5. 2026Personalize Your Large Vision-language Models With In-context Prompt Tuningcited by 3
  6. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoningcited by 2
  7. 2026A2TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clippingcited by 2
  8. 2025STARE at the Structure: Steering ICL Exemplar Selection with Structural Alignmentcited by 1
  9. 2026HIFICL: High-Fidelity In-Context Learning for Multimodal Taskscited by 1
  10. 2026Balancing Multimodal Domain Generalization via Gradient Modulation and Projectioncited by 1
  11. 2026Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learningcited by 1
  12. 2026Beyond Surface Imitation: Contrastive Modeling for Reasoning Path Alignment in Multimodal In-Context Learningcited by 1
  13. 2025ModRWKV: Transformer Multimodality in Linear Timecited by 0
  14. 2025Zero-shot Cross-lingual NER via Mitigating Language Difference: An Entity-aligned Translation Perspectivecited by 0
  15. 2025Two Heads are Better than One: Distilling Large Language Model Features Into Small Models with Feature Decomposition and Mixturecited by 0
  16. 2026UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learningcited by 0
  17. 2026Neutral-Reference Prompting for Vision-Language Modelscited by 0
  18. 2026Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Modelscited by 0
  19. 2026Grounding latent algorithm routing in transformer reasoningcited by 0
  20. 2026When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICLcited by 0
  21. 2026Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learningcited by 0
  22. 2026Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptationscited by 0
  23. 2026Two Vectors Replace In-Context Demos: Structured Task Adaptation via Embeddingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  2. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  3. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  4. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
  5. 2023Towards More Unified In-context Visual Understanding
  6. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning
  7. 2025Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning
  8. 2026UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy