# TADS: Task-Aware Data Selection for Multi-Task Multimodal Pre-Training

arXiv: https://arxiv.org/abs/2602.05251 · HTML: https://mlsanity.com/paper/2602.05251/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 34 references that are in the map.
- First posted on arXiv 5 Feb 2026.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [Data Filtering Networks](https://mlsanity.com/paper/2309.17425/) (2023)
- [T-MARS: Improving Visual Representations by Circumventing Text Feature Learning](https://mlsanity.com/paper/2307.03132/) (2023)
- [MMRL: Multi-Modal Representation Learning for Vision-Language Models](https://mlsanity.com/paper/2503.08497/) (2025)
- [Sieve: Multimodal Dataset Pruning Using Image Captioning Models](https://mlsanity.com/paper/2310.02110/) (2023)
- [The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer](https://mlsanity.com/paper/2504.10462/) (2025)
- [TSDS: Data Selection for Task-Specific Model Finetuning](https://mlsanity.com/paper/2410.11303/) (2024)
- [Optimizing ML Training with Metagradient Descent](https://mlsanity.com/paper/2503.13751/) (2025)
- [CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning](https://mlsanity.com/paper/2405.19547/) (2024)
- [The Devil is in the Details: A Deep Dive into the Rabbit Hole of Data Filtering](https://mlsanity.com/paper/2309.15954/) (2023)
- [HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts](https://mlsanity.com/paper/2404.17507/) (2024)
- [CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization](https://mlsanity.com/paper/2510.17847/) (2025)
- [Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning](https://mlsanity.com/paper/2506.09473/) (2025)
- [Quality over Quantity: Boosting Data Efficiency Through Ensembled Multimodal Data Curation](https://mlsanity.com/paper/2502.08211/) (2025)
- [Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Models](https://mlsanity.com/paper/2510.03903/) (2025)
- [Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining](https://mlsanity.com/paper/2503.08805/) (2025)
- [Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVA](https://mlsanity.com/paper/2505.06356/) (2025)
- [Context-Adaptive Multi-Prompt Embedding with Large Language Models for Vision-Language Alignment](https://mlsanity.com/paper/2508.02762/) (2025)

## Abstract (arXiv)

Large-scale multimodal pre-trained models like CLIP rely heavily on high-quality training data, yet raw web-crawled datasets are often noisy, misaligned, and redundant, leading to inefficient training and suboptimal generalization. Existing data selection methods are either heuristic-based, suffering from bias and limited diversity, or data-driven but task-agnostic, failing to optimize for multi-task scenarios. To address these gaps, we introduce TADS (Task-Aware Data Selection), a novel framework for multi-task multimodal pre-training that integrates Intrinsic Quality, Task Relevance, and Distributional Diversity into a learnable value function. TADS employs a comprehensive quality assessment system with unimodal and cross-modal operators, quantifies task relevance via interpretable similarity vectors, and optimizes diversity through cluster-based weighting. A feedback-driven meta-learning mechanism adaptively refines the selection strategy based on proxy model performance across multiple downstream tasks. Experiments on CC12M demonstrate that TADS achieves superior zero-shot performance on benchmarks like ImageNet, CIFAR-100, MS-COCO, and Flickr30K, using only 36% of the data while outperforming baselines by an average of 1.0%. This highlights that TADS significantly enhances data efficiency by curating a high-utility subset that yields a much higher performance ceiling within the same computational constraints.