mlsanityData to 14 Oct 2026

Demystifying CLIP Data

arXiv 2309.16671 ↗First posted 28 Sep 2023cs.CV cs.CLcited by 270

Not on the map. Why?

Cited by 106 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) is an approach that has advanced research and applications in computer vision, fueling modern recognition systems and generative models. We believe that the main ingredient to the success of CLIP is its data and not the model architecture or pre-training objective. However, CLIP only provides very limited information about its data and how it has been collected, leading to works that aim to reproduce CLIP's data by filtering with its model parameters. In this work, we intend to reveal CLIP's data curation approach and in our pursuit of making it open to the community introduce Metadata-Curated Language-Image Pre-training (MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's concepts) and yields a balanced subset over the metadata distribution. Our experimental study rigorously isolates the model and training settings, concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M image-text data pairs outperforms CLIP's data on multiple standard benchmarks. In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy, surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining the same training budget, attains 72.4%. Our observations hold across various model sizes, exemplified by ViT-H achieving 80.5%, without any bells-and-whistles. Curation code and training data distribution on metadata is made available at https://github.com/facebookresearch/MetaCLIP.

Key facts. Cited by 270 papers in the map (data through 14 Oct 2026). Builds on 10 of its 17 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

10 of 17 references.

References · 10

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  6. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  7. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  8. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  9. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  10. 2023CiT: Curation in Training for Effective Vision-Language Datacited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 89; 2025: 93; 2026: 85.

270 papers.

Sort:

By area:

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  2. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  3. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  4. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  5. 2023Data Filtering Networkscited by 250
  6. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  7. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  8. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  9. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 138
  10. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 124
  11. 2025MiMo-VL Technical Reportcited by 115
  12. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 113
38 more
  1. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 113
  2. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 111
  3. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 89
  4. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
  5. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 84
  6. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 83
  7. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 70
  8. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  9. 2025Scaling Language-Free Visual Representation Learningcited by 66
  10. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  11. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 64
  12. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 58
  13. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  14. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  15. 2024Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approachcited by 50
  16. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 48
  17. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 47
  18. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretrainingcited by 42
  19. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 41
  20. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 40
  21. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 39
  22. 2024OpenStreetView-5M: The Many Roads to Global Visual Geolocationcited by 38
  23. 2024Diffusion Feedback Helps CLIP See Bettercited by 38
  24. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  25. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  26. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 34
  27. 2024LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivitycited by 33
  28. 2024Data curation via joint example selection further accelerates multimodal learningcited by 33
  29. 2024Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidancecited by 30
  30. 2025AToken: A Unified Tokenizer for Visioncited by 30
  31. 2024Meta-Prompting for Automating Zero-shot Visual Recognition with LLMscited by 29
  32. 2024UNIC: Universal Classification Models via Multi-teacher Distillationcited by 27
  33. 2024Garment3DGen: 3D Garment Stylization and Texture Generationcited by 26
  34. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Modelscited by 26
  35. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  36. 2025LongCat-Flash-Omni Technical Reportcited by 26
  37. 2024Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filterscited by 25
  38. 2024Towards Multimodal In-Context Learning for Vision & Language Modelscited by 25

50 most cited of 270 All 270 (JSON)