Corrupted but Not Broken: Understanding and Mitigating the Negative Impacts of Corrupted Data in Visual Instruction Tuning
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Visual Instruction Tuning (VIT) aims to enhance Multimodal Large Language Models (MLLMs), yet its effectiveness is often compromised by corrupted datasets with issues such as hallucinated content, incorrect responses, and poor OCR quality. Previous approaches to address these challenges have focused on refining datasets through high-quality data collection or rule-based filtering that can be costly or limited in scope. In this paper, we conduct a systematic investigation into the impact of corrupted data on MLLMs and discover that, although corrupted data degrade model performance, such adverse effects are largely reversible, and MLLMs are bf corrupted but not broken. Specifically, we find that disabling a small subset of parameters can almost fully restore performance. Moreover, corrupted MLLMs inherently possess the capability to differentiate between clean and corrupted samples, facilitating dataset cleaning without external intervention. Building on these insights, we introduce a corruption-robust training paradigm that significantly surpasses existing strategies for mitigating the effects of corrupted data.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 63 references that are in the map. First posted on arXiv 18 Feb 2025.
Builds on
38 of 63 references.
Closest ancestors · 8
Same direction, newest first
- 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
- 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning19 Dec
- 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
- 2023Improved Baselines with Visual Instruction Tuning5 Oct
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
- 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
Other references · 30
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
- 2019Towards VQA Models That Can Readcited by 1,696
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
- 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
- 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
- 2017A Closer Look at Memorization in Deep Networkscited by 936
18 more
- 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
- 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
- 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
- 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
- 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
- 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
- 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
- 2018Joint Optimization Framework for Learning with Noisy Labelscited by 293
- 2020Combating noisy labels by agreement: A joint training method with co-regularizationcited by 289
- 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
- 2019Probabilistic End-to-end Noise Correction for Learning with Noisy Labelscited by 210
- 2020A Survey of Label-noise Representation Learning: Past, Present and Futurecited by 121
- 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
- 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
- 2021MultiSiam: Self-supervised Multi-instance Siamese Representation Learning for Autonomous Drivingcited by 41
- 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 34
- 2019Searching to Exploit Memorization Effect in Learning from Corrupted Labelscited by 11
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Evian: Towards Explainable Visual Instruction-tuning Data Auditing
- 2025MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuning
- 2023Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator
- 2025Visual Compositional Tuning
- 2024Clean Evaluations on Contaminated Visual Language Models
- 2024Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks
- 2025Picking the Cream of the Crop: Visual-Centric Data Selection with Collaborative Agents
- 2025Visual Instruction Bottleneck Tuning