mlsanityData to 14 Oct 2026

Mimic In-Context Learning for Multimodal Tasks

arXiv 2504.08851 ↗First posted 11 Apr 2025cs.LG cs.AIcited by 12

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recently, In-context Learning (ICL) has become a significant inference paradigm in Large Multimodal Models (LMMs), utilizing a few in-context demonstrations (ICDs) to prompt LMMs for new tasks. However, the synergistic effects in multimodal data increase the sensitivity of ICL performance to the configurations of ICDs, stimulating the need for a more stable and general mapping function. Mathematically, in Transformer-based models, ICDs act as "shift vectors" added to the hidden states of query tokens. Inspired by this, we introduce Mimic In-Context Learning (MimIC) to learn stable and generalizable shift effects from ICDs. Specifically, compared with some previous shift vector-based methods, MimIC more strictly approximates the shift effects by integrating lightweight learnable modules into LMMs with four key enhancements: 1) inserting shift vectors after attention layers, 2) assigning a shift vector to each attention head, 3) making shift magnitude query-dependent, and 4) employing a layer-wise alignment loss. Extensive experiments on two LMMs (Idefics-9b and Idefics2-8b-base) across three multimodal tasks (VQAv2, OK-VQA, Captioning) demonstrate that MimIC outperforms existing shift vector-based methods. The code is available at https://github.com/Kamichanw/MimIC.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 62 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

55 of 62 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  4. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning1 Jul
  5. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning21 Jun
  6. 2024Parrot: Multilingual Visual Instruction Tuning4 Jun
  7. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning1 Oct
  8. 2023A Survey on Multimodal Large Language Models23 Jun
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  10. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
33 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2022A Survey on In-context Learningcited by 721
  4. 2018Object Hallucination in Image Captioningcited by 669
  5. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  6. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  7. 2024What matters when building vision-language models?cited by 326
  8. 2023In-Context Learning Creates Task Vectorscited by 312
  9. 2023Function Vectors in Large Language Modelscited by 308
  10. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  11. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
  12. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  13. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  14. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  15. 2023Compositional Exemplars for In-context Learningcited by 144
  16. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  17. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  18. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  19. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70
  20. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  21. 2024What Makes Multimodal In-Context Learning Work?cited by 56
  22. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  23. 2023Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizercited by 53
  24. 2021Reordering Examples Helps during Priming-based Few-Shot Learningcited by 50
  25. 2023Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-trainingcited by 37
  26. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 33
  27. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selectioncited by 20
  28. 2024Dataset Regeneration for Sequential Recommendationcited by 19
  29. 2024How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?cited by 13
  30. 2024DG-PIC: Domain Generalized Point-In-Context Learning for Point Cloud Understandingcited by 13
  31. 2024Scaling New Frontiers: Insights into Large Recommendation Modelscited by 13
  32. 2024Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropycited by 9
  33. 2024LIVE: Learnable In-Context Vector for Visual Question Answeringcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 15
  2. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  3. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 7
  4. 2026Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformerscited by 5
  5. 2025Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learningcited by 4
  6. 2025Safe Semantics, Unsafe Interpretations: Tackling Implicit Reasoning Safety in Large Vision-Language Modelscited by 3
  7. 2025ERF-BA-TFD+: A Multimodal Model for Audio-Visual Deepfake Detectioncited by 1
  8. 2026HIFICL: High-Fidelity In-Context Learning for Multimodal Taskscited by 1
  9. 2026Parallel In-context Learning for Large Vision Language Modelscited by 1
  10. 2026Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathologycited by 1
  11. 2026Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learningcited by 1
  12. 2026When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks
  2. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  3. 2024Adapting Large Multimodal Models to Distribution Shifts: The Role of In-Context Learning
  4. 2026Beyond Surface Imitation: Contrastive Modeling for Reasoning Path Alignment in Multimodal In-Context Learning
  5. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
  6. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  7. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering
  8. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning