mlsanityData to 7 Oct 2026

Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection

arXiv 2402.12501 ↗First posted 19 Feb 2024cs.CLcited by 48

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Data selection in instruction tuning emerges as a pivotal process for acquiring high-quality data and training instruction-following large language models (LLMs), but it is still a new and unexplored research area for vision-language models (VLMs). Existing data selection approaches on LLMs either rely on single unreliable scores, or use downstream tasks for selection, which is time-consuming and can lead to potential over-fitting on the chosen evaluation datasets. To address this challenge, we introduce a novel dataset selection method, Self-Filter, that utilizes the VLM itself as a filter. This approach is inspired by the observation that VLMs benefit from training with the most challenging instructions. Self-Filter operates in two stages. In the first stage, we devise a scoring network to evaluate the difficulty of training instructions, which is co-trained with the VLM. In the second stage, we use the trained score net to measure the difficulty of each instruction, select the most challenging samples, and penalize similar samples to encourage diversity. Comprehensive experiments on LLaVA and MiniGPT-4 show that Self-Filter can reach better results compared to full data settings with merely about 15% samples, and can achieve superior performance against competitive baselines.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 47 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

24 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets23 Aug
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  5. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  8. 2023LIMA: Less Is More for Alignmentcited by 1,080
  9. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  10. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  11. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  12. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
6 more
  1. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  2. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  3. 2023Instruction Mining: Instruction Data Selection for Tuning Large Language Modelscited by 57
  4. 2023One-Shot Learning as Instruction Data Prospector for Large Language Modelscited by 43
  5. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignmentcited by 33
  6. 2023AlpaGasus: Training A Better Alpaca with Fewer Datacited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 23; 2026: 14.

48 papers.

Sort:

By area:

  1. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  2. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  3. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selectioncited by 57
  4. 2024ICONS: Influence Consensus for Vision-Language Data Selectioncited by 28
  5. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  6. 2024A Watermark for Order-Agnostic Language Modelscited by 14
  7. 2025Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuningcited by 12
  8. 2024De-mark: Watermark Removal in Large Language Modelscited by 11
  9. 2025VisionTS++: Cross-Modal Time Series Foundation Model with Continual Pre-trained Vision Backbonescited by 11
  10. 2024Distortion-free Watermarks are not Truly Distortion-free under Watermark Key Collisionscited by 10
  11. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimizationcited by 10
  12. 2025MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuningcited by 9
36 more
  1. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Miningcited by 8
  2. 2025CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimizationcited by 8
  3. 2024Curriculum Learning with Quality-Driven Data Selectioncited by 7
  4. 2025Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectoriescited by 7
  5. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Modelscited by 7
  6. 2025MotionEdit: Benchmarking and Learning Motion-Centric Image Editingcited by 7
  7. 2026VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuningcited by 6
  8. 2024Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captionscited by 5
  9. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Modelscited by 5
  10. 2026ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuningcited by 5
  11. 2025Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMscited by 4
  12. 2025Learning What Matters: Prioritized Concept Learning via Relative Error-driven Sample Selectioncited by 3
  13. 2026SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Modelscited by 3
  14. 2024Less is More: High-value Data Selection for Visual Instruction Tuningcited by 2
  15. 2024Error-driven Data-efficient Large Multimodal Model Tuningcited by 2
  16. 2025OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Modelscited by 2
  17. 2025Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Modelscited by 2
  18. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alonecited by 2
  19. 2025Visual Compositional Tuningcited by 1
  20. 2025Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVAcited by 1
  21. 2025Exploring Instruction Data Quality for Explainable Image Quality Assessmentcited by 1
  22. 2026Does the Question Really Matter? Training-Free Data Selection for Vision-Language SFTcited by 1
  23. 2026DataProphet: Demystifying Supervision Data Generalization in Multimodal LLMscited by 1
  24. 2026DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Modelscited by 1
  25. 2026Evian: Towards Explainable Visual Instruction-tuning Data Auditingcited by 1
  26. 2026Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuningcited by 1
  27. 2025CL3DOR: Contrastive Learning for 3D Large Multimodal Models via Odds Ratio on High-Resolution Point Cloudscited by 0
  28. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Qualitycited by 0
  29. 2025SafeFix: Targeted Model Repair via Controlled Image Generationcited by 0
  30. 2025Delta-AttnMask: Attention-Guided Masked Hidden States for Efficient Data Selection and Augmentationcited by 0
  31. 2025Uncovering Scaling Laws for Large Language Models via Inverse Problemscited by 0
  32. 2026CAST: Collapse-Aware multi-Scale Topology Fusion for Multimodal Coreset Selectioncited by 0
  33. 2026MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-Language Modelscited by 0
  34. 2026Mapping the Concept Landscape: Structural Perception of Global Distributions for Transparent Data Pruningcited by 0
  35. 2026BIRD: Distilling Decision Boundaries into Rationales for MLLM Adaptationcited by 0
  36. 2026Scalable, Transferable Meta-network for Data Selection Requires a Different Loss (and Why the Obvious Choice is Problematic)cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Vision-Language Instruction Tuning: A Review and Analysis
  2. 2024Rethinking Overlooked Aspects in Vision-Language Models
  3. 2025Instruction-Following Evaluation of Large Vision-Language Models
  4. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models
  5. 2026Data Selection Through Iterative Self-Filtering for Vision-Language Settings
  6. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
  7. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning
  8. 2025Cream of the Crop: Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuning