mlsanityData to 14 Oct 2026

TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration

arXiv 2505.17098 ↗First posted 21 May 2025cs.CL cs.CVcited by 23

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal in-context learning (ICL) has emerged as a key mechanism for harnessing the capabilities of large vision-language models (LVLMs). However, its effectiveness remains highly sensitive to the quality of input ICL sequences, particularly for tasks involving complex reasoning or open-ended generation. A major limitation is our limited understanding of how LVLMs actually exploit these sequences during inference. To bridge this gap, we systematically interpret multimodal ICL through the lens of task mapping, which reveals how local and global relationships within and among demonstrations guide model reasoning. Building on this insight, we present TACO, a lightweight transformer-based model equipped with task-aware attention that dynamically configures ICL sequences. By injecting task-mapping signals into the autoregressive decoding process, TACO creates a bidirectional synergy between sequence construction and task reasoning. Experiments on five LVLMs and nine datasets demonstrate that TACO consistently surpasses baselines across diverse ICL tasks. These results position task mapping as a novel and valuable perspective for interpreting and improving multimodal ICL.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 61 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

34 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Multimodal Language Models See Better When They Look Shallower30 Apr
  2. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering6 Apr
  3. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations5 Mar
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec

Other references · 30

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  11. 2022A Survey on In-context Learningcited by 721
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
18 more
  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  2. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  3. 2023Larger language models do in-context learning differentlycited by 394
  4. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  5. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  6. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  7. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70
  8. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70
  9. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  10. 2024An Information-Theoretic Analysis of In-Context Learningcited by 40
  11. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 36
  12. 2023An Early Evaluation of GPT-4V(ision)cited by 31
  13. 2023In-Context Demonstration Selection with Cross Entropy Differencecited by 17
  14. 2024Learning to Retrieve Iteratively for In-Context Learningcited by 13
  15. 2024In-Context Learning Demonstration Selection via Influence Analysiscited by 11
  16. 2024Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanismcited by 6
  17. 2025OptiSeq: Ordering Examples On-The-Fly for In-Context Learningcited by 6
  18. 2023Comparable Demonstrations are Important in In-Context Learning: A Novel Perspective on Demonstration Selectioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 10.

23 papers.

Sort:

By area:

  1. 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigationcited by 23
  2. 2025CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learningcited by 18
  3. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 16
  4. 2026TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrievalcited by 15
  5. 2025SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysiscited by 9
  6. 2025MRFD: Multi-Region Fusion Decoding with Self-Consistency for Mitigating Hallucinations in LVLMscited by 9
  7. 2025From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoningcited by 9
  8. 2025FakeSV-VLM: Taming VLM for Detecting Fake Short-Video News via Progressive Mixture-Of-Experts Adaptercited by 7
  9. 2025Faithful-First Reasoning, Planning, and Acting for Multimodal LLMscited by 7
  10. 2026ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulationcited by 6
  11. 2026Seeing Further and Wider: Joint Spatio-Temporal Enlargement for Micro-Video Popularity Predictioncited by 5
  12. 2025Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust Text-based Person Retrievalcited by 4
11 more
  1. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 2
  2. 2025ReLoop: "Seeing Twice and Thinking Backwards" via Closed-loop Training to Mitigate Hallucinations in Multimodal understandingcited by 1
  3. 2025STARE at the Structure: Steering ICL Exemplar Selection with Structural Alignmentcited by 1
  4. 2025Beyond Function-Level Search: Repository-Aware Dual-Encoder Code Retrieval with Adversarial Verificationcited by 1
  5. 2026HIFICL: High-Fidelity In-Context Learning for Multimodal Taskscited by 1
  6. 2026Balancing Multimodal Domain Generalization via Gradient Modulation and Projectioncited by 1
  7. 2025Zero-shot Cross-lingual NER via Mitigating Language Difference: An Entity-aligned Translation Perspectivecited by 0
  8. 2026UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learningcited by 0
  9. 2026Gradient-free Task-Conditioned Retrieval for On-Device In-Context Learningcited by 0
  10. 2026MAG: MAnifold Guided Semi-Supervised Multi-modal In-Context Learningcited by 0
  11. 2026MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  2. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
  3. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  4. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  5. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
  6. 2023Towards More Unified In-context Visual Understanding
  7. 2025CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMs
  8. 2024What Makes Multimodal In-Context Learning Work?