# Curriculum Learning with Quality-Driven Data Selection

arXiv: https://arxiv.org/abs/2407.00102 · HTML: https://mlsanity.com/paper/2407.00102/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding](https://mlsanity.com/paper/2306.17107/) (2023)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [RedCaps: web-curated image-text data created by the people, for the people](https://mlsanity.com/paper/2111.11431/) (2021)
- [SVIT: Scaling up Visual Instruction Tuning](https://mlsanity.com/paper/2307.04087/) (2023)
- [M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning](https://mlsanity.com/paper/2306.04387/) (2023)
- [Improving Multimodal Datasets with Image Captioning](https://mlsanity.com/paper/2307.10350/) (2023)
- [MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets](https://mlsanity.com/paper/2308.12067/) (2023)
- [Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection](https://mlsanity.com/paper/2402.12501/) (2024)
- [One-Shot Learning as Instruction Data Prospector for Large Language Models](https://mlsanity.com/paper/2312.10302/) (2023)
- [Curriculum Self-Paced Learning for Cross-Domain Object Detection](https://mlsanity.com/paper/1911.06849/) (2019)
- [StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data](https://mlsanity.com/paper/2308.10253/) (2023)
- [Curriculum-based Deep Reinforcement Learning for Quantum Control](https://mlsanity.com/paper/2012.15427/) (2020)

## Built on by (most cited first, 7 of 7)

- [The Best Instruction-Tuning Data are Those That Fit](https://mlsanity.com/paper/2502.04194/) (2025)
- [Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples](https://mlsanity.com/paper/2502.09650/) (2025)
- [Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness](https://mlsanity.com/paper/2412.06293/) (2024)
- [Small-Margin Preferences Still Matter-If You Train Them Right](https://mlsanity.com/paper/2602.00954/) (2026)
- [Delta-AttnMask: Attention-Guided Masked Hidden States for Efficient Data Selection and Augmentation](https://mlsanity.com/paper/2508.09199/) (2025)
- [SceneParser: Hierarchical Scene Parsing for Visual Semantics Understanding](https://mlsanity.com/paper/2605.14923/) (2026)
- [DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes](https://mlsanity.com/paper/2607.24516/) (2026)

## Abstract (arXiv)

The impressive multimodal capabilities demonstrated by OpenAI's GPT-4 have generated significant interest in the development of Multimodal Large Language Models (MLLMs). Visual instruction tuning of MLLMs with machine-generated instruction-following data has shown to enhance zero-shot capabilities across various tasks. However, there has been limited exploration into controlling the quality of the instruction data.Current methodologies for data selection in MLLMs often rely on single, unreliable scores or use downstream tasks for selection, which is time-consuming and can lead to potential overfitting on the chosen evaluation datasets. To mitigate these limitations, we propose a novel data selection methodology that utilizes image-text correlation and model perplexity to evaluate and select data of varying quality. This approach leverages the distinct distribution of these two attributes, mapping data quality into a two-dimensional space that allows for the selection of data based on their location within this distribution. By utilizing this space, we can analyze the impact of task type settings, used as prompts, on data quality. Additionally, this space can be used to construct multi-stage subsets of varying quality to facilitate curriculum learning. Our research includes comprehensive experiments conducted on various datasets. The results emphasize substantial enhancements in five commonly assessed capabilities compared to using the complete dataset. Our codes, data, and models are publicly available at: https://anonymous.4open.science/r/EHIT-31B4