mlsanityData to 7 Oct 2026

Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Models

arXiv 2510.00040 ↗First posted 27 Sep 2025cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large vision-language models (VLMs) achieve strong benchmark performance, but controlling their behavior through instruction tuning remains difficult. Reducing the budget of instruction tuning dataset often causes regressions, as heuristic strategies treat models as black boxes and overlook the latent capabilities that govern learning. We introduce Capability-Attributed Data Curation (CADC), a framework that shifts curation from task-specific heuristics to intrinsic capability analysis. CADC discovers intrinsic capabilities in an unsupervised manner from gradient-based learning trajectories, attributes training data to these capabilities via influence estimation, and curates capability-aware curricula through balanced selection and staged sequencing. This transforms black-box instruction tuning into a controllable, capability-driven process. With as little as 5% of the original data, CADC surpasses full-data training on multimodal benchmarks. These results validate intrinsic capabilities as the fundamental building blocks of model learning and establish CADC as a principle paradigm for instruction data curation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 40 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

32 of 40 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024ICONS: Influence Consensus for Vision-Language Data Selection31 Dec
  2. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  3. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Models16 Jun
  4. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  5. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection19 Feb
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  9. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  10. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  11. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  12. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
11 more
  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  3. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  4. 2018From Louvain to Leiden: guaranteeing well-connected communitiescited by 379
  5. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  6. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  7. 2023Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasetscited by 18
  8. 2025Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Modelscited by 16
  9. 2024Law of the Weakest Link: Cross Capabilities of Large Language Modelscited by 12
  10. 2019An Optimization Framework for Task Sequencing in Curriculum Learningcited by 7
  11. 2021Heuristic Search Planning with Deep Neural Networks using Imitation, Attention and Curriculum Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026GRACE: Gradient-aligned Reasoning Data Curation for Efficient Post-trainingcited by 0
  2. 2026RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Semantic Capability Acquisition and Specialization During Vision-Language Model Fine-Tuning
  2. 2026DICS: Exploring Data Intrinsic Consistency for Visual Instruction Selection
  3. 2024Less is More: High-value Data Selection for Visual Instruction Tuning
  4. 2025Learning What Matters: Prioritized Concept Learning via Relative Error-driven Sample Selection
  5. 2025Data Metabolism: An Efficient Data Design Schema For Vision Language Model
  6. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  7. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases
  8. 2026SkillRater: Untangling Capabilities in Multimodal Data