mlsanityData to 7 Oct 2026

Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models

arXiv 2605.08145 ↗First posted 3 May 2026cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Current vision language models face hallucination and robustness issues against ambiguous or corrupted modalities. We hypothesize that these issues can be addressed by exploiting the shared information between modalities to compensate for the impaired one. To this end, we analyze multimodal interactions -- redundant (shared), unique (exclusive), and synergistic (emergent) task-relevant information provided by the modalities -- to determine their impacts on model reliability. Specifically, amplifying redundant interactions would increase this exploitable shared information to resolve these issues; yet, modern instruction datasets often eliminate redundancies to prioritize visual grounding. We bridge this gap through a self-captioning workflow featuring a textscMultimodal Interaction Gate: a mechanism to convert unique interactions into redundant interactions. Our findings suggest that increasing redundancy can reduce visual induced errors by 38.3% and improve consistency by 16.8%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 45 references that are in the map. First posted on arXiv 3 May 2026.

Builds on

36 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  2. 2023VILA: On Pre-training for Visual Language Models12 Dec

Other references · 34

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  9. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  10. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  11. 2024What matters when building vision-language models?cited by 321
  12. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
22 more
  1. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  2. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  3. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 130
  4. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
  5. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  6. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  7. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 62
  8. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  9. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 30
  10. 2025I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Expertscited by 29
  11. 2021A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decompositioncited by 25
  12. 2022A Differential Entropy Estimator for Training Neural Networkscited by 20
  13. 2023Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applicationscited by 20
  14. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decompositioncited by 16
  15. 2024Measuring Cross-Modal Interactions in Multimodal Modelscited by 16
  16. 2016Measuring multivariate redundant information with pointwise common change in surprisalcited by 13
  17. 2024Quantifying Knowledge Distillation Using Partial Information Decompositioncited by 12
  18. 2023Multimodal Fusion Interactions: A Study of Human and Automatic Quantificationcited by 9
  19. 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?cited by 8
  20. 2023DocMSU: A Comprehensive Benchmark for Document-level Multimodal Sarcasm Understandingcited by 7
  21. 2025Efficient Quantification of Multimodal Interaction at Sample Levelcited by 5
  22. 2025Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement
  2. 2026I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models
  3. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidance
  4. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  5. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models
  6. 2023On the Performance of Multimodal Language Models
  7. 2026Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement
  8. 2026Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models