mlsanityData to 7 Oct 2026

An Inverse Scaling Law for CLIP Training

arXiv 2305.07017 ↗First posted 11 May 2023cs.CVcited by 76

Not on the map. Why?

Cited by 30 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

CLIP, one of the pioneering foundation models that connect images and text, has enabled many recent breakthroughs in computer vision. However, its associated training cost is prohibitively high, imposing a significant barrier to its widespread exploration. In this paper, we present a surprising finding that there exists an inverse scaling law for CLIP training, whereby the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training. Moreover, we showcase that the strategy for reducing image/text token length plays a crucial role in determining the quality of this scaling law. As a result of this finding, we are able to successfully train CLIP even with limited computational resources. For example, using 8 A100 GPUs, our CLIP models achieve zero-shot top-1 ImageNet-1k accuracies of 63.2% in ~2 days, 67.8% in ~3 days, and 69.3% in ~4 days. Our method also works well when scaling up -- with G/14, we register a new record of 83.0% ImageNet-1k zero-shot accuracy, and meanwhile accelerate the training by ~33x compared to its OpenCLIP counterpart. By reducing the computation barrier associated with CLIP, we hope to inspire more research in this field, particularly from academics. Our code is available at https://github.com/UCSC-VLAA/CLIPA.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 66 of its 73 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

66 of 73 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2023GPT-4 Technical Reportcited by 18,132
  9. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  10. 2022Training language models to follow instructions with human feedbackcited by 15,712
  11. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  12. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2022A ConvNet for the 2020scited by 3,545
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  13. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  14. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  15. 2019Natural Adversarial Examplescited by 1,401
  16. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  17. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  18. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  19. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  20. 2017Mixed Precision Trainingcited by 1,135
  21. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  22. 2021Scaling Vision Transformerscited by 993
  23. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  24. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  25. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  26. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  27. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  28. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  29. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  30. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  31. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  32. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  33. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  34. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  35. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  36. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  37. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  38. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 33; 2025: 25; 2026: 9.

76 papers.

Sort:

By area:

  1. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  2. 2024Pixtral 12Bcited by 174
  3. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  4. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  5. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  6. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  7. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  8. 2025RewardDance: Reward Scaling in Visual Generationcited by 63
  9. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 55
  10. 2024ViTamin: Designing Scalable Vision Models in the Vision-Language Eracited by 39
  11. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  12. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 38
38 more
  1. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  2. 2024Effective pruning of web-scale datasets based on complexity of concept clusterscited by 34
  3. 2024Revisiting Adversarial Training at Scalecited by 33
  4. 2024Data curation via joint example selection further accelerates multimodal learningcited by 32
  5. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scalingcited by 31
  6. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 28
  7. 2024Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculumcited by 25
  8. 2023CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a 10,000 Budget; An Extra 4,000 Unlocks 81.8% Accuracycited by 23
  9. 2024MoDE: CLIP Data Experts via Clusteringcited by 22
  10. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  11. 2024An Empirical Study Into What Matters for Calibrating Vision-Language Modelscited by 20
  12. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 20
  13. 2023Rejuvenating image-GPT as Strong Visual Representation Learnerscited by 16
  14. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignmentcited by 15
  15. 2025OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learningcited by 15
  16. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14
  17. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  18. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  19. 2025CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classificationcited by 12
  20. 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Modelscited by 12
  21. 2023DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanationscited by 11
  22. 2024Classification Done Right for Vision-Language Pre-Trainingcited by 10
  23. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustnesscited by 10
  24. 2024Seeing the Unseen: Visual Common Sense for Semantic Placementcited by 9
  25. 2024LG-CAV: Train Any Concept Activation Vector with Language Guidancecited by 9
  26. 2024Scaling White-Box Transformers for Visioncited by 8
  27. 2023An Empirical Study of Scaling Law for OCRcited by 7
  28. 2024FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resourcescited by 7
  29. 2024Dissecting Representation Misalignment in Contrastive Learning via Influence Functioncited by 7
  30. 2025GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysiscited by 7
  31. 2025Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognitioncited by 7
  32. 2025Seeing What Matters: Empowering CLIP with Patch Generation-to-Selectioncited by 7
  33. 2025OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learningcited by 7
  34. 2024Exploring the Design Space of Visual Context Representation in Video MLLMscited by 6
  35. 2024SCAN: Bootstrapping Contrastive Pre-training for Data Efficiencycited by 6
  36. 2025Advancing Compositional Awareness in CLIP with Efficient Fine-Tuningcited by 5
  37. 2026OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generationcited by 5
  38. 2025What Holds Back Open-Vocabulary Segmentation?cited by 4

50 most cited of 76 All 76 (JSON)