# Data curation via joint example selection further accelerates multimodal learning

arXiv: https://arxiv.org/abs/2406.17711 · HTML: https://mlsanity.com/paper/2406.17711/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 58 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [Data Filtering Networks](https://mlsanity.com/paper/2309.17425/) (2023)
- [Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt](https://mlsanity.com/paper/2206.07137/) (2022)
- [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://mlsanity.com/paper/2307.06304/) (2023)
- [Working hard to know your neighbor's margins: Local descriptor learning loss](https://mlsanity.com/paper/1705.10872/) (2017)
- [FlexiViT: One Model for All Patch Sizes](https://mlsanity.com/paper/2212.08013/) (2022)
- [Rho-1: Not All Tokens Are What You Need](https://mlsanity.com/paper/2404.07965/) (2024)

## Built on by (most cited first, 30 of 33)

- [Model Predictive Task Sampling for Efficient and Robust Adaptation](https://mlsanity.com/paper/2501.11039/) (2025)
- [MobileCLIP2: Improving Multi-Modal Reinforced Training](https://mlsanity.com/paper/2508.20691/) (2025)
- [Active Data Curation Effectively Distills Large-Scale Multimodal Models](https://mlsanity.com/paper/2411.18674/) (2024)
- [Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection](https://mlsanity.com/paper/2410.10636/) (2024)
- [ELIP: Enhanced Visual-Language Foundation Models for Image Retrieval](https://mlsanity.com/paper/2502.15682/) (2025)
- [Learning from Streaming Video with Orthogonal Gradients](https://mlsanity.com/paper/2504.01961/) (2025)
- [Influential Language Data Selection via Gradient Trajectory Pursuit](https://mlsanity.com/paper/2410.16710/) (2024)
- [Data Lineage Inference: Uncovering Privacy Vulnerabilities of Dataset Pruning](https://mlsanity.com/paper/2411.15796/) (2024)
- [MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation](https://mlsanity.com/paper/2501.04155/) (2025)
- [A Bayesian Approach to Data Point Selection](https://mlsanity.com/paper/2411.03768/) (2024)
- [Context-Aware Multimodal Pretraining](https://mlsanity.com/paper/2411.15099/) (2024)
- [Fair Foundation Models for Medical Image Analysis: Challenges and Perspectives](https://mlsanity.com/paper/2502.16841/) (2025)
- [AdaDeDup: Adaptive Hybrid Data Pruning for Efficient Large-Scale Object Detection Training](https://mlsanity.com/paper/2507.00049/) (2025)
- [EGM: Efficient Visual Grounding Language Models](https://mlsanity.com/paper/2601.13633/) (2026)
- [Learnability-Guided Diffusion for Dataset Distillation](https://mlsanity.com/paper/2604.00519/) (2026)
- [Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling](https://mlsanity.com/paper/2501.13779/) (2025)
- [OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models](https://mlsanity.com/paper/2507.19262/) (2025)
- [Layer-Aware Influence for Online Data Valuation Estimation](https://mlsanity.com/paper/2510.16007/) (2025)
- [Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws](https://mlsanity.com/paper/2505.06699/) (2025)
- [Autoguided Online Data Curation for Diffusion Model Training](https://mlsanity.com/paper/2509.15267/) (2025)
- [Dynamic Jointly Batch Selection for Data Efficient Machine Translation Fine-Tuning](https://mlsanity.com/paper/2511.04406/) (2025)
- [Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluation](https://mlsanity.com/paper/2412.07429/) (2024)
- [Optimizing Data Curation through Spectral Analysis and Joint Batch Selection (SALN)](https://mlsanity.com/paper/2412.17069/) (2024)
- [Efficient Training of Deep Networks using Guided Spectral Data Selection: A Step Toward Learning What You Need](https://mlsanity.com/paper/2507.04269/) (2025)
- [Online Data Curation for Object Detection via Marginal Contributions to Dataset-level Average Precision](https://mlsanity.com/paper/2511.14197/) (2025)
- [ReCCur: A Recursive Corner-Case Curation Framework for Robust Vision-Language Understanding in Open and Edge Scenarios](https://mlsanity.com/paper/2601.03011/) (2026)
- [Xuanwu: Evolving General Multimodal Models into an Industrial-Grade Foundation for Content Ecosystems](https://mlsanity.com/paper/2603.29211/) (2026)
- [Hierarchical Pre-Training of Vision Encoders with Large Language Model](https://mlsanity.com/paper/2604.00086/) (2026)
- [SLAP: Stratified Loss-based Pruning for On-Policy Data-Efficient Instruction Tuning](https://mlsanity.com/paper/2605.23969/) (2026)
- [Data Selection Through Iterative Self-Filtering for Vision-Language Settings](https://mlsanity.com/paper/2606.23611/) (2026)

## Abstract (arXiv)

Data curation is an essential component of large-scale pretraining. In this work, we demonstrate that jointly selecting batches of data is more effective for learning than selecting examples independently. Multimodal contrastive objectives expose the dependencies between data and thus naturally yield criteria for measuring the joint learnability of a batch. We derive a simple and tractable algorithm for selecting such batches, which significantly accelerate training beyond individually-prioritized data points. As performance improves by selecting from larger super-batches, we also leverage recent advances in model approximation to reduce the associated computational overhead. As a result, our approach--multimodal contrastive learning with joint example selection (JEST)--surpasses state-of-the-art models with up to 13× fewer iterations and 10× less computation. Essential to the performance of JEST is the ability to steer the data selection process towards the distribution of smaller, well-curated datasets via pretrained reference models, exposing the level of data curation as a new dimension for neural scaling laws.