mlsanityData to 7 Oct 2026

Scaling Pre-training to One Hundred Billion Data for Vision Language Models

arXiv 2502.07617 ↗First posted 11 Feb 2025cs.CVcited by 18

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We provide an empirical investigation of the potential of pre-training vision-language models on an unprecedented scale: 100 billion examples. We find that model performance tends to saturate at this scale on many common Western-centric classification and retrieval benchmarks, such as COCO Captions. Nevertheless, tasks of cultural diversity achieve more substantial gains from the 100-billion scale web data, thanks to its coverage of long-tail concepts. Furthermore, we analyze the model's multilinguality and show gains in low-resource languages as well. In addition, we observe that reducing the size of the pretraining dataset via quality filters like using CLIP, typically used to enhance performance, may inadvertently reduce the cultural diversity represented in large-scale datasets. Our results highlight that while traditional benchmarks may not benefit significantly from scaling noisy, raw web data to 100 billion examples, this data scale is vital for building truly inclusive multimodal systems.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 82 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

62 of 82 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Scalable Vision Language Model Training via High Quality Data Curation10 Jan
  2. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  3. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models22 May
  4. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic10 Apr
  5. 2024The Neglected Tails in Vision-Language Models23 Jan
  6. 2023PaLI-3 Vision Language Models: Smaller, Faster, Stronger13 Oct
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2013Describing Textures in the Wildcited by 2,110
38 more
  1. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  6. 2021Scaling Vision Transformerscited by 993
  7. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  8. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  9. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  11. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  12. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  13. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  14. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  15. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  16. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  17. 2021Explaining Neural Scaling Lawscited by 353
  18. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  19. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  20. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  21. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  22. 2023Data Filtering Networkscited by 247
  23. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  24. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  25. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  26. 2020Characterising Bias in Compressed Modelscited by 162
  27. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
  28. 2021Scaling Laws for Neural Machine Translationcited by 110
  29. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  30. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  31. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 87
  32. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80
  33. 2021Learning Curve Theorycited by 79
  34. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  35. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 51
  36. 2023GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognitioncited by 50
  37. 2015How much data is needed to train a medical image deep learning system to achieve necessary high accuracy?cited by 45
  38. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 6.

18 papers.

Sort:

By area:

  1. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  2. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
  3. 2025Scaling Language-Free Visual Representation Learningcited by 62
  4. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 59
  5. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  6. 2025Dynamic Classifier-Free Diffusion Guidance via Online Feedbackcited by 11
  7. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  8. 2025Scalable Strategies for Continual Learning with Replaycited by 3
  9. 2025DinoAtten3D: Slice-Level Attention Aggregation of DinoV2 for 3D Brain MRI Anomaly Classificationcited by 3
  10. 2025NeuCLIP: Efficient Large-Scale CLIP Training with Neural Normalizer Optimizationcited by 3
  11. 2026Towards Next-Generation LLM Training: From the Data-Centric Perspectivecited by 3
  12. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Trainingcited by 1
6 more
  1. 2025GRAPE: Let GRPO Supervise Query Rewriting by Ranking for Retrievalcited by 1
  2. 2026Shaping Schema via Language Representation as the Next Frontier for LLM Intelligence Expandingcited by 1
  3. 2026Language-Instructed Vision Embeddings for Controllable and Generalizable Perceptioncited by 1
  4. 2026DiD It in 87 Minutes: A Label-Free Softmax-to-Linear Adaptation of Vision Transformers for Object Detectioncited by 1
  5. 2026Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understandingcited by 0
  6. 2026Contrastive Action-Image Pre-training for Visuomotor Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  2. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  3. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models
  4. 2026Beyond Language Modeling: An Exploration of Multimodal Pretraining
  5. 2024M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks
  6. 2023A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation
  7. 2025A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling
  8. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledge