mlsanityData to 7 Oct 2026

HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks

arXiv 2603.12760 ↗First posted 13 Mar 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In-Context Learning (ICL) is a significant paradigm for Large Multimodal Models (LMMs), using a few in-context demonstrations (ICDs) for new task adaptation. However, its performance is sensitive to demonstration configurations and computationally expensive. Mathematically, the influence of these demonstrations can be decomposed into a dynamic mixture of the standard attention output and the context values. Current approximation methods simplify this process by learning a "shift vector". Inspired by the exact decomposition, we introduce High-Fidelity In-Context Learning (HIFICL) to more faithfully model the ICL mechanism. HIFICL consists of three key components: 1) a set of "virtual key-value pairs" to act as a learnable context, 2) a low-rank factorization for stable and regularized training, and 3) a simple end-to-end training objective. From another perspective, this mechanism constitutes a form of context-aware Parameter-Efficient Fine-Tuning (PEFT). Extensive experiments show that HiFICL consistently outperforms existing approximation methods on several multimodal benchmarks. The code is available at https://github.com/bbbandari/HiFICL.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 48 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

43 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025True Multimodal In-Context Learning Needs Attention to the Visual Context21 Jul
  2. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration21 May
  3. 2025Mimic In-Context Learning for Multimodal Tasks11 Apr
  4. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering6 Apr
  5. 2023Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models15 Dec
  6. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?29 Nov
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  8. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
23 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  3. 2022In-context Learning and Induction Headscited by 930
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  6. 2018Object Hallucination in Image Captioningcited by 661
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  8. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  9. 2024What matters when building vision-language models?cited by 321
  10. 2023In-Context Learning Creates Task Vectorscited by 302
  11. 2023Function Vectors in Large Language Modelscited by 292
  12. 2023Towards Generalist Biomedical AIcited by 286
  13. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  14. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  15. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  16. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  17. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 31
  18. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  19. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  20. 2023Towards Understanding How Transformers Learn In-context Through a Representation Learning Lenscited by 15
  21. 2025Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learningcited by 10
  22. 2025Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attentioncited by 8
  23. 2025Detecting Data Contamination in LLMs via In-Context Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy
  2. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
  3. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
  4. 2025Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning
  5. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks
  6. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations
  7. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
  8. 2025What do vision-language models see in the context? Investigating multimodal in-context learning