mlsanityData to 14 Oct 2026

Meta CLIP 2: A Worldwide Scaling Recipe

arXiv 2507.22062 ↗First posted 29 Jul 2025cs.CV cs.CLcited by 64

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Contrastive Language-Image Pretraining (CLIP) is a popular foundation model, supporting from zero-shot classification, retrieval to encoders for multimodal large language models (MLLMs). Although CLIP is successfully trained on billion-scale image-text pairs from the English world, scaling CLIP's training further to learning from the worldwide web data is still challenging: (1) no curation method is available to handle data points from non-English world; (2) the English performance from existing multilingual CLIP is worse than its English-only counterpart, i.e., "curse of multilinguality" that is common in LLMs. Here, we present Meta CLIP 2, the first recipe training CLIP from scratch on worldwide web-scale image-text pairs. To generalize our findings, we conduct rigorous ablations with minimal changes that are necessary to address the above challenges and present a recipe enabling mutual benefits from English and non-English world data. In zero-shot ImageNet classification, Meta CLIP 2 ViT-H/14 surpasses its English-only counterpart by 0.8% and mSigLIP by 0.7%, and surprisingly sets new state-of-the-art without system-level confounding factors (e.g., translation, bespoke architecture changes) on multilingual benchmarks, such as CVQA with 57.4%, Babel-ImageNet with 50.2% and XM3600 with 64.3% on image-to-text retrieval.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 67 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

52 of 67 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  8. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  9. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  10. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  13. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  14. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  15. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  16. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 309
  17. 2018Learning Word Vectors for 157 Languagescited by 280
  18. 2023Demystifying CLIP Datacited by 270
  19. 2023Data Filtering Networkscited by 250
  20. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  21. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  22. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 153
  23. 2019Does Object Recognition Work for Everyone?cited by 151
  24. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 120
  25. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  26. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  27. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  28. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  29. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  30. 2025Scaling Language-Free Visual Representation Learningcited by 66
  31. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  32. 2023GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognitioncited by 50
  33. 2020Towards Zero-shot Cross-lingual Image Retrievalcited by 25
  34. 2023NLLB-CLIP -- train performant multilingual image retrieval model on a budgetcited by 24
  35. 2025Scaling Pre-training to One Hundred Billion Data for Vision Language Modelscited by 19
  36. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Modelscited by 16
  37. 2019Graph-RISE: Graph-Regularized Image Semantic Embeddingcited by 15
  38. 2024Multilingual Diversity Improves Vision-Language Representationscited by 14

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 50.

64 papers.

Sort:

By area:

  1. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 50
  2. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 39
  3. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 13
  4. 2025In Pursuit of Pixel Supervision for Visual Pre-trainingcited by 13
  5. 2026FireRed-OCR Technical Reportcited by 13
  6. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 13
  7. 2026OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligencecited by 12
  8. 2026The Information Geometry of Softmax: Probing and Steeringcited by 11
  9. 2025Brought a Gun to a Knife Fight: Modern VFM Baselines Outgun Specialized Detectors on In-the-Wild AI Image Detectioncited by 10
  10. 2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
  11. 2025SuperFlow: Training Flow Matching Models with RL on the Flycited by 10
  12. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 10
38 more
  1. 2026WorldVQA: Measuring Atomic World Knowledge in Multimodal Large Language Modelscited by 8
  2. 2026OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generationcited by 5
  3. 2026Sapiens2cited by 4
  4. 2026ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledgecited by 3
  5. 2026Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Visioncited by 2
  6. 2026Let ViT Speak: Generative Language-Image Pre-trainingcited by 2
  7. 2025On the dynamic evolution of CLIP texture-shape bias and its relationship to human alignment and model robustnesscited by 1
  8. 2025GRAPE: Let GRPO Supervise Query Rewriting by Ranking for Retrievalcited by 1
  9. 2025Language as a Label: Zero-Shot Multimodal Classification of Everyday Postures under Data Scarcitycited by 1
  10. 2025Text-Guided Semantic Image Encodercited by 1
  11. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Trainingcited by 1
  12. 2025EchoVLM: Measurement-Grounded Multimodal Learning for Echocardiographycited by 1
  13. 2026HEDGE: Heterogeneous Ensemble for Detection of AI-GEnerated Images in the Wildcited by 1
  14. 2026LOGER: Local--Global Ensemble for Robust Deepfake Detection in the Wildcited by 1
  15. 2026CRAFT: Clinical Reward-Aligned Finetuning for Medical Image Synthesiscited by 1
  16. 2026Decoupling the Declarative from the Procedural in Vision-Language-Action Modelscited by 1
  17. 2026On the Limitations of Cross-Lingual Consistency in Multilingual Text-to-image Generationcited by 1
  18. 2025Eigenfunction Extraction for Ordered Representation Learningcited by 0
  19. 2025Learning Ordinal Degradation Representations with Textual Priors for Diffusion-Based Blind Image Super-Resolutioncited by 0
  20. 2026DanQing: An Up-to-Date Large-Scale Chinese Vision-Language Pre-training Datasetcited by 0
  21. 2026ObjEmbed: Towards Universal Multimodal Object Embeddingscited by 0
  22. 2026Xray-Visual Models: Scaling Vision models on Industry Scale Datacited by 0
  23. 2026PinPoint: Evaluation of Composed Image Retrieval with Explicit Negatives, Multi-Image Queries, and Paraphrase Testingcited by 0
  24. 2026When Surfaces Lie: Exploiting Wrinkle-Induced Attention Shift to Attack Vision-Language Modelscited by 0
  25. 2026SearchAD: Large-Scale Rare Image Retrieval Dataset for Autonomous Drivingcited by 0
  26. 2026Boosting Robust AIGI Detection with LoRA-based Pairwise Trainingcited by 0
  27. 2026Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understandingcited by 0
  28. 2026CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Modelscited by 0
  29. 2026What Matters for Grocery Product Retrieval with Open Source Vision Language Modelscited by 0
  30. 2026Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domaincited by 0
  31. 2026RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representationcited by 0
  32. 2026ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIPcited by 0
  33. 2026Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoderscited by 0
  34. 2026AdaBoosting Text Prompts for Vision-Language Modelscited by 0
  35. 2026DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detectioncited by 0
  36. 2026Fine-Grained Food Image Understanding via Target-Aware Data Alignmentcited by 0
  37. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Biascited by 0
  38. 2026Enhancing VLM Reward Models Through Structure-Aware Fine-Tuningcited by 0

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images
  2. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese
  3. 2023CLIP-KD: An Empirical Study of CLIP Model Distillation
  4. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  5. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining
  6. 2024Diffusion Feedback Helps CLIP See Better
  7. 2025FG-CLIP: Fine-Grained Visual and Textual Alignment
  8. 2024Toward a Holistic Evaluation of Robustness in CLIP Models