mlsanityData to 14 Oct 2026

Concept-skill Transferability-based Data Selection for Large Vision-Language Models

arXiv 2406.10995 ↗First posted 16 Jun 2024cs.CV cs.LGcited by 24

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Instruction tuning, or supervised finetuning on extensive task-specific data, is necessary for Large Vision-Language Models (LVLMs) to generalize well across a broad range of vision-language (VL) tasks. However, training on large VL datasets can become prohibitively expensive. In this work, we introduce COINCIDE, an effective and scalable data selection technique that uses a small model as a reference model to select visual instruction tuning data for efficient finetuning of a target LVLM, focusing on diversity and transferability. Specifically, we cluster the training data using internal activations from a small model, which identifies VL concept-skill compositions needed by a target LVLM. We then sample data from these diverse clusters by considering their density and transferability, or the ability to transfer well to other concept-skill compositions. This approach ensures the diversity of these compositions, which is vital for LVLM generalization. Extensive experiments demonstrate that COINCIDE achieves superior performance and data selection efficiency against 8 strong baselines on two distinct datasets: LLaVA-1.5 and Vision-Flan. Using only 20% of the LLaVA-1.5 dataset, COINCIDE achieves performance comparable to the LVLM finetuned on the whole dataset, with 70% reduction of the wall-clock running time. On the Vision-Flan dataset, our method achieves superior results with only 16.7% of the training data.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 61 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

45 of 61 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  2. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  3. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  4. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection19 Feb
  5. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning18 Feb
  6. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  7. 2023Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers13 Nov
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets23 Aug
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  8. 2023LIMA: Less Is More for Alignmentcited by 1,100
  9. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  10. 2019Gradient based sample selection for online continual learningcited by 571
  11. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  12. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
21 more
  1. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  2. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  3. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  4. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  5. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  6. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
  7. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  8. 2023The Quantization Model of Neural Scalingcited by 138
  9. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  10. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 109
  11. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  12. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  13. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 69
  14. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  15. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  16. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 44
  17. 2019The Information Complexity of Learning Tasks, their Structure and their Distancecited by 36
  18. 2024Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Modelscited by 32
  19. 2024Visual Concept Connectome (VCC): Open World Concept Discovery and their Interlayer Connections in Deep Modelscited by 22
  20. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biasescited by 13
  21. 2017Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Taskscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 8.

24 papers.

Sort:

By area:

  1. 2024ICONS: Influence Consensus for Vision-Language Data Selectioncited by 29
  2. 2025Vision Function Layer in Multimodal LLMscited by 17
  3. 2025Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencodercited by 12
  4. 2025Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuningcited by 12
  5. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativenesscited by 10
  6. 2024Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selectioncited by 9
  7. 2025From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibrationcited by 8
  8. 2025CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimizationcited by 8
  9. 2026VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuningcited by 8
  10. 2025Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectoriescited by 7
  11. 2025VideoTG-R1: Boosting Video Temporal Grounding via Curriculum Reinforcement Learning on Reflected Boundary Annotationscited by 7
  12. 2025Learning What Matters: Prioritized Concept Learning via Relative Error-driven Sample Selectioncited by 3
12 more
  1. 2025Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Modelscited by 2
  2. 2026On the Difficulty of Learning a Meta-network for Training Data Selectioncited by 2
  3. 2025Cream of the Crop: Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuningcited by 1
  4. 2025Visual Compositional Tuningcited by 1
  5. 2025Data-Efficient Ensemble Weather Forecasting with Diffusion Modelscited by 1
  6. 2025Exploring Instruction Data Quality for Explainable Image Quality Assessmentcited by 1
  7. 2026Does the Question Really Matter? Training-Free Data Selection for Vision-Language SFTcited by 1
  8. 2026Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuningcited by 1
  9. 2026Is One-Shot In-Context Learning Helpful for Data Selection in Task-Specific Fine-Tuning of Multimodal LLMs?cited by 0
  10. 2026GRACE: Gradient-aligned Reasoning Data Curation for Efficient Post-trainingcited by 0
  11. 2026Mapping the Concept Landscape: Structural Perception of Global Distributions for Transparent Data Pruningcited by 0
  12. 2026Scalable, Transferable Meta-network for Data Selection Requires a Different Loss (and Why the Obvious Choice is Problematic)cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Less is More: High-value Data Selection for Visual Instruction Tuning
  2. 2024Rethinking Overlooked Aspects in Vision-Language Models
  3. 2024From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning
  4. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection
  5. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  6. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models
  7. 2025Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring
  8. 2024From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning