mlsanityData to 14 Oct 2026

Reproducible scaling laws for contrastive language-image learning

arXiv 2212.07143 ↗First posted 14 Dec 2022cs.LG cs.AI cs.CVcited by 1,173

Not on the map. Why?

Cited by 307 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data & models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. Our large-scale experiments involve models trained on up to two billion image-text pairs and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We find that the training distribution plays a key role in scaling laws as the OpenAI and OpenCLIP models exhibit different scaling behavior despite identical model architectures and similar training recipes. We open-source our evaluation workflow and all models, including the largest public CLIP models, to ensure reproducibility and make scaling laws research more accessible. Source code and instructions to reproduce this study will be available at https://github.com/LAION-AI/scaling-laws-openclip

Key facts. Cited by 1,173 papers in the map (data through 14 Oct 2026). Builds on 62 of its 90 references that are in the map. First posted on arXiv 14 Dec 2022.

Builds on

62 of 90 references.

References · 62

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  13. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  14. 2019Natural Adversarial Examplescited by 1,408
  15. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  16. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  17. 2021Scaling Vision Transformerscited by 999
  18. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  19. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  20. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  21. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  22. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  23. 2021Robust fine-tuning of zero-shot modelscited by 797
  24. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  25. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  26. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  27. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  28. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  29. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  30. 2018Do Better ImageNet Models Transfer Better?cited by 565
  31. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  32. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  33. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  34. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  35. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  36. 2020Are we done with ImageNet?cited by 333
  37. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
  38. 2015YFCC100M: The New Data in Multimedia Researchcited by 235

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 147; 2024: 334; 2025: 379; 2026: 313.

1,173 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  4. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  5. 2023A Survey on Multimodal Large Language Modelscited by 909
  6. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  8. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  9. 2023LERF: Language Embedded Radiance Fieldscited by 543
  10. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  11. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  12. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
38 more
  1. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  2. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 363
  3. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 337
  4. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  5. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  6. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  7. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  8. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  9. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  10. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  11. 2023Demystifying CLIP Datacited by 270
  12. 2023Data Filtering Networkscited by 250
  13. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  14. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  15. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  16. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 219
  17. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  18. 2023Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robustcited by 187
  19. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  20. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 165
  21. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 155
  22. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  23. 2024Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Modelscited by 142
  24. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  25. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  26. 2023APRIL-GAN: A Zero-/Few-Shot Anomaly Classification and Segmentation Method for CVPR 2023 VAND Workshop Challenge Tracks 1&2: 1st Place on Zero-shot AD and 4th Place on Few-shot ADcited by 130
  27. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  28. 2025Phantom: Subject-consistent video generation via cross-modal alignmentcited by 129
  29. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  30. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  31. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
  32. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 120
  33. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  34. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  35. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 117
  36. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  37. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 113
  38. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111

50 most cited of 1,173 All 1,173 (JSON)