mlsanityData to 14 Oct 2026

ICONS: Influence Consensus for Vision-Language Data Selection

arXiv 2501.00654 ↗First posted 31 Dec 2024cs.CV cs.CL cs.LGcited by 29

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Training vision-language models via instruction tuning relies on large data mixtures spanning diverse tasks and domains, yet these mixtures frequently include redundant information that increases computational costs without proportional gains. Existing methods typically rely on task-agnostic heuristics to estimate data importance, limiting their effectiveness across tasks. We introduce ICONS, a gradient-based Influence CONsensus approach for vision-language data Selection. Our method leverages first-order training dynamics to estimate each example's influence on validation performance, then aggregates these estimates across tasks via majority voting. This cross-task consensus identifies consistently valuable data points while mitigating score calibration and outlier sensitivity, enabling robust and scalable data selection for diverse multitask mixtures. Models trained on our selected 20% data subset from LLAVA-665K (respectively: from CAMBRIAN-7M, from VISION-FLAN-186K) retain 98.6% (respectively: 98.8%, 99.8%) of full-dataset performance. We demonstrate that our selected data generalizes to unseen tasks and model architectures, and release three compact subsets LLAVA-ICONS-133K, CAMBRIAN-ICONS-1.4M, and VISION-FLAN-ICONS-37K for efficient vision-language model development.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 52 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

45 of 52 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  2. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Models16 Jun
  3. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection19 Feb
  4. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning18 Feb
  5. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark22 Jan
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  12. 2019Towards VQA Models That Can Readcited by 1,718
24 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  5. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  6. 2016A Diagram Is Worth A Dozen Imagescited by 804
  7. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  8. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  9. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  10. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  11. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  12. 2021InfographicVQAcited by 437
  13. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  14. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  15. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  16. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  17. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  18. 2022Training Data Influence Analysis and Estimation: A Surveycited by 135
  19. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 133
  20. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  21. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  22. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 69
  23. 2024Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinementcited by 13
  24. 2024Influential Language Data Selection via Gradient Trajectory Pursuitcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 16.

29 papers.

Sort:

By area:

  1. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selectioncited by 61
  2. 2025CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimizationcited by 8
  3. 2026VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuningcited by 8
  4. 2025Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectoriescited by 7
  5. 2026Motion Attribution for Video Generationcited by 5
  6. 2026ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuningcited by 5
  7. 2026Can Generalist Agents Automate Data Curation?cited by 5
  8. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4
  9. 2025Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weightscited by 3
  10. 2025Learning What Matters: Prioritized Concept Learning via Relative Error-driven Sample Selectioncited by 3
  11. 2025Differential-informed Sample Selection Accelerates Multimodal Contrastive Learningcited by 3
  12. 2025Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Modelscited by 2
17 more
  1. 2025Visual Compositional Tuningcited by 1
  2. 2025Concepts or Skills? Rethinking Instruction Selection for Multi-modal Modelscited by 1
  3. 2025Data-Efficient Ensemble Weather Forecasting with Diffusion Modelscited by 1
  4. 2025Exploring Instruction Data Quality for Explainable Image Quality Assessmentcited by 1
  5. 2026Does the Question Really Matter? Training-Free Data Selection for Vision-Language SFTcited by 1
  6. 2026DataProphet: Demystifying Supervision Data Generalization in Multimodal LLMscited by 1
  7. 2026DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Modelscited by 1
  8. 2026Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuningcited by 1
  9. 2025Delta-AttnMask: Attention-Guided Masked Hidden States for Efficient Data Selection and Augmentationcited by 0
  10. 2026GRACE: Gradient-aligned Reasoning Data Curation for Efficient Post-trainingcited by 0
  11. 2026MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-Language Modelscited by 0
  12. 2026Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Trainingcited by 0
  13. 2026Hierarchical Data Selection via Manifold Coverage and Sparse Feature Coverage in LLM Post-trainingcited by 0
  14. 2026Mapping the Concept Landscape: Structural Perception of Global Distributions for Transparent Data Pruningcited by 0
  15. 2026DICS: Exploring Data Intrinsic Consistency for Visual Instruction Selectioncited by 0
  16. 2026Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuningcited by 0
  17. 2026Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Less is More: High-value Data Selection for Visual Instruction Tuning
  2. 2024Rethinking Overlooked Aspects in Vision-Language Models
  3. 2026Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain
  4. 2023Towards Efficient Vision-Language Tuning: More Information Density, More Generalizability
  5. 2025Test-Time Consistency in Vision Language Models
  6. 2023Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models
  7. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases
  8. 2025Mordal: Automated Pretrained Model Selection for Vision Language Models