mlsanityData to 7 Oct 2026

Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning

arXiv 2406.15334 ↗First posted 21 Jun 2024cs.CV cs.AI cs.CLcited by 49

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The recent success of interleaved Large Multimodal Models (LMMs) in few-shot learning suggests that in-context learning (ICL) with many examples can be promising for learning new tasks. However, this many-shot multimodal ICL setting has one crucial problem: it is fundamentally limited by the model's context length set at pretraining. The problem is especially prominent in the multimodal domain, which processes both text and images, requiring additional tokens. This motivates the need for a multimodal method to compress many shots into fewer tokens without finetuning. In this work, we enable LMMs to perform multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors (MTV) -- compact implicit representations of in-context examples compressed in the model's attention heads. Specifically, we first demonstrate the existence of such MTV in LMMs and then leverage these extracted MTV to enable many-shot in-context learning for various vision-and-language tasks. Our experiments suggest that MTV can scale in performance with the number of compressed shots and generalize to similar out-of-domain tasks without additional context length for inference. Code: https://github.com/Brandon3964/MultiModal-Task-Vector

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 86 of its 104 references that are in the map. First posted on arXiv 21 Jun 2024.

Builds on

86 of 104 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Towards Multimodal In-Context Learning for Vision & Language Models19 Mar
  2. 2023Generative Multimodal Models are In-Context Learners20 Dec
  3. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  6. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun

Other references · 80

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  13. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  14. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  15. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  16. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  17. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  18. 2015VQA: Visual Question Answeringcited by 2,489
  19. 2022Emergent Abilities of Large Language Modelscited by 2,483
  20. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  21. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  22. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  23. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  24. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  25. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  26. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  27. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  28. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  29. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  30. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  31. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  32. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  33. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  34. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  35. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  36. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  37. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  38. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 24; 2026: 19.

49 papers.

Sort:

By area:

  1. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  2. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  3. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  4. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  5. 2025Analyzing Finetuning Representation Shift for Multimodal LLMs Steeringcited by 15
  6. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  7. 2024Vision-Language Models Create Cross-Modal Task Representationscited by 14
  8. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
  9. 2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
  10. 2025Mimic In-Context Learning for Multimodal Taskscited by 12
  11. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Featurescited by 11
  12. 2025Do different prompting methods yield a common task representation in language models?cited by 9
37 more
  1. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 9
  2. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  3. 2025Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitationscited by 8
  4. 2024ELICIT: LLM Augmentation via External In-Context Capabilitycited by 7
  5. 2025Efficient Multi-modal Long Context Learning for Training-free Adaptationcited by 7
  6. 2025Just-in-time and distributed task representations in language modelscited by 7
  7. 2024Teaching VLMs to Localize Specific Objects from In-context Examplescited by 6
  8. 2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
  9. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrationscited by 5
  10. 2026Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformerscited by 5
  11. 2025Unlocking In-Context Learning for Natural Datasets Beyond Language Modellingcited by 4
  12. 2025Adaptive Task Vectors for Large Language Modelscited by 4
  13. 2025Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learningcited by 4
  14. 2025MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderationcited by 3
  15. 2025Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillationcited by 3
  16. 2026Retrieving Counterfactuals Improves Visual In-Context Learningcited by 3
  17. 2025FinLMM-R1: Enhancing Financial Reasoning in LMM through Scalable Data and Reward Designcited by 2
  18. 2026Enhancing Large Language Models for Time-Series Forecasting via Vector-Injected In-Context Learningcited by 1
  19. 2026ArtifactLens: Hundreds of Labels Are Enough for Artifact Detection with VLMscited by 1
  20. 2026Differentially Private Multimodal In-Context Learningcited by 1
  21. 2026Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selectioncited by 1
  22. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 1
  23. 2026Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generationcited by 1
  24. 2025Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answeringcited by 0
  25. 2025Variational Task Vector Compositioncited by 0
  26. 2025Implicit Federated In-context Learning For Task-Specific LLM Fine-Tuningcited by 0
  27. 2026CALM: Class-Conditional Sparse Attention Vectors for Large Audio-Language Modelscited by 0
  28. 2026Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learningcited by 0
  29. 2026Global Logic and Local Search: Dual-Stream Multimodal In-Context Learning for Verifiable Industrial Anomaly Detectioncited by 0
  30. 2026Gradient-free Task-Conditioned Retrieval for On-Device In-Context Learningcited by 0
  31. 2026In-Context Collapse in Vision-Language Models and How to Mitigate it?cited by 0
  32. 2026HyperFix: Combinatorial Nonlinear Correction for Task Vector Mergingcited by 0
  33. 2026When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICLcited by 0
  34. 2026Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learningcited by 0
  35. 2026MAIL: Memory-driven, Adaptive, Incremental, and Literature-grounded Framework for Hypothesis Generation in Chemistrycited by 0
  36. 2026Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptationscited by 0
  37. 2026Capturing In-Context Learning Dynamics with Task Operatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FewMMBench: A Benchmark for Multimodal Few-Shot Learning
  2. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering
  3. 2024Context-Aware Multimodal Pretraining
  4. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  5. 2024What Makes Multimodal In-Context Learning Work?
  6. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
  7. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
  8. 2023Link-Context Learning for Multimodal LLMs