mlsanityData to 14 Oct 2026

DataComp: In search of the next generation of multimodal datasets

arXiv 2304.14108 ↗First posted 27 Apr 2023cs.CV cs.CL cs.LGcited by 644

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms. To address this shortcoming in the ML ecosystem, we introduce DataComp, a testbed for dataset experiments centered around a new candidate pool of 12.8 billion image-text pairs from Common Crawl. Participants in our benchmark design new filtering techniques or curate new data sources and then evaluate their new dataset by running our standardized CLIP training code and testing the resulting model on 38 downstream test sets. Our benchmark consists of multiple compute scales spanning four orders of magnitude, which enables the study of scaling trends and makes the benchmark accessible to researchers with varying resources. Our baseline experiments show that the DataComp workflow leads to better training sets. In particular, our best baseline, DataComp-1B, enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet, outperforming OpenAI's CLIP ViT-L/14 by 3.7 percentage points while using the same training procedure and compute. We release DataComp and all accompanying code at www.datacomp.ai.

Key facts. Cited by 644 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 101 of its 183 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

101 of 183 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 100

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2022A ConvNet for the 2020scited by 3,579
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2017Billion-scale similarity search with GPUscited by 2,497
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  13. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  14. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  15. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  16. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  17. 2019Natural Adversarial Examplescited by 1,408
  18. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  19. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  20. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  21. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  22. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  23. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  24. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  25. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  26. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  27. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  28. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  29. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  30. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  31. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  32. 2018Do Better ImageNet Models Transfer Better?cited by 565
  33. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  34. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  35. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  36. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
  37. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  38. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422

50 most cited of 100 All 100 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 77; 2024: 224; 2025: 205; 2026: 138.

644 papers.

Sort:

By area:

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  5. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  6. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  7. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  9. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  10. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  11. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  12. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
38 more
  1. 2024OmniGen: Unified Image Generationcited by 394
  2. 2025Kimi-VL Technical Reportcited by 388
  3. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  4. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  5. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  6. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  7. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  8. 2023Demystifying CLIP Datacited by 270
  9. 2023Data Filtering Networkscited by 250
  10. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  11. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  12. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  13. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  14. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  15. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  16. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  17. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  18. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  19. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  20. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 175
  21. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  22. 2023Raising the Bar of AI-generated Image Detection with CLIPcited by 164
  23. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 161
  24. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  25. 2023Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Modelscited by 157
  26. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  27. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  28. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 132
  29. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  30. 2023Teaching Arithmetic to Small Transformerscited by 123
  31. 20234M: Massively Multimodal Masked Modelingcited by 120
  32. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 119
  33. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  34. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  35. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  36. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  37. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  38. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111

50 most cited of 644 All 644 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Models
  2. 2026MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset
  3. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
  4. 2026DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models
  5. 2026i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models
  6. 2025D3G: Diverse Demographic Data Generation Increases Zero-Shot Image Classification Accuracy within Multimodal Models
  7. 2024Diffusion Feedback Helps CLIP See Better
  8. 2024GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Models