mlsanityData to 7 Oct 2026

Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters

arXiv 2403.02677 ↗First posted 5 Mar 2024cs.CV cs.CLcited by 25

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We propose a novel framework for filtering image-text data by leveraging fine-tuned Multimodal Language Models (MLMs). Our approach outperforms predominant filtering methods (e.g., CLIPScore) via integrating the recent advances in MLMs. We design four distinct yet complementary metrics to holistically measure the quality of image-text data. A new pipeline is established to construct high-quality instruction data for fine-tuning MLMs as data filters. Comparing with CLIPScore, our MLM filters produce more precise and comprehensive scores that directly improve the quality of filtered data and boost the performance of pre-trained models. We achieve significant improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2) and various downstream tasks. Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore. An additional ablation study is provided to verify our design choices for the MLM filter.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 48 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

35 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  5. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets23 Aug

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
18 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  3. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  4. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  6. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  7. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  8. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  9. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  10. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  11. 2023Demystifying CLIP Datacited by 265
  12. 2023Data Filtering Networkscited by 247
  13. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  14. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  15. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  16. 2023Mass-Producing Failures of Multimodal Systems with Language Modelscited by 36
  17. 2023The Devil is in the Details: A Deep Dive into the Rabbit Hole of Data Filteringcited by 19
  18. 2022Visually-Augmented Language Modelingcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 10; 2026: 3.

25 papers.

Sort:

By area:

  1. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  2. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  3. 2024CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learningcited by 20
  4. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  5. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  6. 2024Multimodal Misinformation Detection by Learning from Synthetic Data with Multimodal LLMscited by 13
  7. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 12
  8. 2025Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resourcescited by 9
  9. 2025CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimizationcited by 8
  10. 2024SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Modelscited by 6
  11. 2024SCAN: Bootstrapping Contrastive Pre-training for Data Efficiencycited by 6
  12. 2025Synthetic Visual Genomecited by 6
13 more
  1. 2025Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMscited by 4
  2. 2025Towards Multimodal Active Learning: Efficient Learning with Limited Paired Datacited by 3
  3. 2024Beyond Filtering: Adaptive Image-Text Quality Enhancement for MLLM Pretrainingcited by 2
  4. 2025Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pixcited by 2
  5. 2025Train a Unified Multimodal Data Quality Classifier with Synthetic Datacited by 2
  6. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alonecited by 2
  7. 2024Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Modelcited by 1
  8. 2026Evian: Towards Explainable Visual Instruction-tuning Data Auditingcited by 1
  9. 2024Data Processing for the OpenGPT-X Model Familycited by 0
  10. 2025SafeFix: Targeted Model Repair via Controlled Image Generationcited by 0
  11. 2025InfiMed-Foundation: Pioneering Advanced Multimodal Medical Models with Compute-Efficient Pre-Training and Multi-Stage Fine-Tuningcited by 0
  12. 2025Turning Adversaries into Allies: Reversing Typographic Attacks for Multimodal E-Commerce Product Retrievalcited by 0
  13. 2026TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
  2. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  3. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
  4. 2025Multimodal Large Language Models for Image, Text, and Speech Data Augmentation: A Survey
  5. 2025Optimizing Multimodal Language Models through Attention-based Interpretability
  6. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  7. 2023Multimodal Large Language Models: A Survey
  8. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images