mlsanityData to 14 Oct 2026

Multimodal Task Interference: A Benchmark and Analysis of History-Target Mismatch in Multimodal LLMs

arXiv 2603.18425 ↗First posted 19 Mar 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Task interference, the performance degradation caused by task switches within a single conversation, has been studied exclusively in text-only settings despite the growing prevalence of multimodal dialogue systems. We introduce a benchmark for evaluating this phenomenon in multimodal LLMs, covering six tasks across text and vision with systematic variation of history-target along three axes: modality mismatch, reasoning mismatch, and answer format mismatch. Experiments on both open-weights and proprietary models reveal that task interference is highly directional: switching from text-only to image-based targets causes severe performance drops, while the reverse transition yields minimal degradation. Interference is further amplified when mismatches co-occur across multiple dimensions, and is driven most strongly by modality differences, followed by answer format, while reasoning requirement shifts cause minimal degradation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 17 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

11 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Multimodal Instruction Tuning with Conditional Mixture of LoRA24 Feb

Other references · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  9. 2019TWEETQA: A Social Media Focused Question Answering Datasetcited by 55
  10. 2024LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational Historycited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
  2. 2025Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models
  3. 2026Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs
  4. 2026What We are Missing in Multimodal LLM Evaluation?
  5. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
  6. 2026Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs
  7. 2024Understanding the Role of LLMs in Multimodal Evaluation Benchmarks
  8. 2026CSR-Bench: A Benchmark for Evaluating the Cross-modal Safety and Reliability of MLLMs