mlsanityData to 14 Oct 2026

Improving CLIP Training with Language Rewrites

arXiv 2305.20088 ↗First posted 31 May 2023cs.CV cs.CL cs.LGcited by 217

Not on the map. Why?

Cited by 90 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) stands as one of the most effective and scalable methods for training transferable vision models using paired image and text data. CLIP models are trained using contrastive loss, which typically relies on data augmentations to prevent overfitting and shortcuts. However, in the CLIP training paradigm, data augmentations are exclusively applied to image inputs, while language inputs remain unchanged throughout the entire training process, limiting the exposure of diverse texts to the same image. In this paper, we introduce Language augmented CLIP (LaCLIP), a simple yet highly effective approach to enhance CLIP training through language rewrites. Leveraging the in-context learning capability of large language models, we rewrite the text descriptions associated with each image. These rewritten texts exhibit diversity in sentence structure and vocabulary while preserving the original key concepts and meanings. During training, LaCLIP randomly selects either the original texts or the rewritten versions as text augmentations for each image. Extensive experiments on CC3M, CC12M, RedCaps and LAION-400M datasets show that CLIP pre-training with language rewrites significantly improves the transfer performance without computation or memory overhead during training. Specifically for ImageNet zero-shot accuracy, LaCLIP outperforms CLIP by 8.2% on CC12M and 2.4% on LAION-400M. Code is available at https://github.com/LijieFan/LaCLIP.

Key facts. Cited by 217 papers in the map (data through 14 Oct 2026). Builds on 44 of its 87 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

44 of 87 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
32 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2013Describing Textures in the Wildcited by 2,125
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  13. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  14. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  15. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  16. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  17. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  18. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  19. 2021MetaICL: Learning to Learn In Contextcited by 454
  20. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  21. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  22. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  23. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  24. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  25. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  26. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  27. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  28. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 124
  29. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  30. 2016Learning Visual N-Grams from Web Datacited by 95
  31. 2016How Transferable are Neural Networks in NLP Applications?cited by 23
  32. 2022Where Should I Spend My FLOPS? Efficiency Evaluations of Visual Pre-training Methodscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 25; 2024: 87; 2025: 71; 2026: 34.

217 papers.

Sort:

By area:

  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
  5. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 95
  6. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  7. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  8. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
  9. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  10. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 73
  11. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  12. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
38 more
  1. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 70
  2. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  3. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  4. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 58
  5. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  6. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  7. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 48
  8. 2024Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negationscited by 46
  9. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 45
  10. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 40
  11. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 39
  12. 2023Visual Instruction Tuning with Polite Flamingocited by 38
  13. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 38
  14. 2024TIPS: Text-Image Pretraining with Spatial awarenesscited by 37
  15. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  16. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  17. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 32
  18. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  19. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  20. 2024LoTLIP: Improving Language-Image Pre-training for Long Text Understandingcited by 27
  21. 2023Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Miningcited by 25
  22. 2024CoMT: Chain-of-Medical-Thought Reduces Hallucination in Medical Report Generationcited by 25
  23. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
  24. 2023Medical Vision Language Pretraining: A surveycited by 24
  25. 2024Would Deep Generative Models Amplify Bias in Future Models?cited by 24
  26. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 22
  27. 2023Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusioncited by 21
  28. 2023MLLMs-Augmented Visual-Language Representation Learningcited by 21
  29. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 21
  30. 2023Mitigating Open-Vocabulary Caption Hallucinationscited by 19
  31. 2024UrbanVLP: Multi-Granularity Vision-Language Pretraining for Urban Socioeconomic Indicator Predictioncited by 19
  32. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  33. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contextscited by 17
  34. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  35. 2024Past, Present, and Future of Sensor-Based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Taskcited by 15
  36. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  37. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15
  38. 2024USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentationcited by 14

50 most cited of 217 All 217 (JSON)