mlsanityData to 14 Oct 2026

Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models

arXiv 2505.19616 ↗First posted 26 May 2025cs.LG cs.AI cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models demonstrate strong performance on multimodal benchmarks, yet often exhibit poor robustness when exposed to spurious modality interference, such as irrelevant text in vision understanding, or irrelevant visual content in question answering. At its core, modality interference refers to cases where spurious signals from non-essential modalities distort model decisions, which we systematically analyze through causal, perturbation-based diagnostic experiments. To address this problem, we propose a unified finetuning framework that combines heuristic and adversarial perturbation-based data augmentation with output-level consistency regularization between original and perturbed inputs. Extensive experiments across image-heavy, text-heavy, and multimodal benchmarks, spanning multiple MLLM architectures and model scales, demonstrate consistent improvements in unimodal robustness and generalization, while improving standard multimodal performance.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 50 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

40 of 50 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Models17 Jun
  4. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective27 Mar
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  6. 2023VILA: On Pre-training for Visual Language Models12 Dec
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  9. 2025Qwen2.5-VL Technical Reportcited by 5,587
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
18 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  3. 2019Towards VQA Models That Can Readcited by 1,718
  4. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  5. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  6. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  7. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  8. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 370
  9. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  10. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  11. 2024Attacking Large Language Models with Projected Gradient Descentcited by 94
  12. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  13. 2022CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Modelscited by 53
  14. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  15. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Modelscited by 28
  16. 2023PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizercited by 15
  17. 2024Zero-Shot Relational Learning for Multimodal Knowledge Graphscited by 3
  18. 2024Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledgecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 15.

19 papers.

Sort:

By area:

  1. 2025Latent Visual Reasoningcited by 104
  2. 2026Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitrationcited by 6
  3. 2026Reinforced Attention Learningcited by 6
  4. 2025When Silence Matters: The Impact of Irrelevant Audio on Text Reasoning in Large Audio-Language Modelscited by 4
  5. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  6. 2025Learning Efficient Guardrails for Compliancecited by 2
  7. 2026Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMscited by 2
  8. 2026When Vision Speaks for Soundcited by 2
  9. 2025Understanding the Effects of Distractors on Reasoning Vision-Language Modelscited by 1
  10. 2026ModelLens: Finding the Best for Your Task from Myriads of Modelscited by 1
  11. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  12. 2026CARPE: Context-Aware Image Representation Prioritization via Ensemble for Large Vision-Language Modelscited by 0
7 more
  1. 2026Q-Tacit: Image Quality Assessment via Latent Visual Reasoningcited by 0
  2. 2026Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videoscited by 0
  3. 2026Sensing Which Modality Matters: Evidence-Gated Regularization for Robust VLA Policiescited by 0
  4. 2026MoGround: Measuring and Mitigating Modality Distraction in Vision-Language Modelscited by 0
  5. 2026Relevant Evidence Decoding for Audio-Visual Hallucination Mitigationcited by 0
  6. 2026Controlled Acquisition and Abstention in Three-Channel Score Conflictscited by 0
  7. 2026Selective Listening: Mechanism-Guided Control of Audio Influence in Large Audio-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict
  2. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  3. 2024Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models
  4. 2026Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing
  5. 2026TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
  6. 2025Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional
  7. 2025Dynamic Modality Scheduling for Multimodal Large Models via Confidence, Uncertainty, and Semantic Consistency
  8. 2025When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models