# Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Models

arXiv: https://arxiv.org/abs/2510.00040 · HTML: https://mlsanity.com/paper/2510.00040/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 27 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020)
- [From Louvain to Leiden: guaranteeing well-connected communities](https://mlsanity.com/paper/1810.08473/) (2018)
- [LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models](https://mlsanity.com/paper/2407.12772/) (2024)
- [SmolVLM: Redefining small and efficient multimodal models](https://mlsanity.com/paper/2504.05299/) (2025)
- [How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition](https://mlsanity.com/paper/2310.05492/) (2023)
- [MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI](https://mlsanity.com/paper/2404.16006/) (2024)
- [Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection](https://mlsanity.com/paper/2402.12501/) (2024)
- [ICONS: Influence Consensus for Vision-Language Data Selection](https://mlsanity.com/paper/2501.00654/) (2024)
- [Concept-skill Transferability-based Data Selection for Large Vision-Language Models](https://mlsanity.com/paper/2406.10995/) (2024)
- [Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets](https://mlsanity.com/paper/2302.13959/) (2023)
- [Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Models](https://mlsanity.com/paper/2503.20807/) (2025)
- [Law of the Weakest Link: Cross Capabilities of Large Language Models](https://mlsanity.com/paper/2409.19951/) (2024)

## Built on by (most cited first, 2 of 2)

- [GRACE: Gradient-aligned Reasoning Data Curation for Efficient Post-training](https://mlsanity.com/paper/2605.13130/) (2026)
- [RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation](https://mlsanity.com/paper/2610.00979/) (2026)

## Abstract (arXiv)

Large vision-language models (VLMs) achieve strong benchmark performance, but controlling their behavior through instruction tuning remains difficult. Reducing the budget of instruction tuning dataset often causes regressions, as heuristic strategies treat models as black boxes and overlook the latent capabilities that govern learning. We introduce Capability-Attributed Data Curation (CADC), a framework that shifts curation from task-specific heuristics to intrinsic capability analysis. CADC discovers intrinsic capabilities in an unsupervised manner from gradient-based learning trajectories, attributes training data to these capabilities via influence estimation, and curates capability-aware curricula through balanced selection and staged sequencing. This transforms black-box instruction tuning into a controllable, capability-driven process. With as little as 5% of the original data, CADC surpasses full-data training on multimodal benchmarks. These results validate intrinsic capabilities as the fundamental building blocks of model learning and establish CADC as a principle paradigm for instruction data curation.