mlsanityData to 7 Oct 2026

Modeling Caption Diversity in Contrastive Vision-Language Pretraining

arXiv 2405.00740 ↗First posted 30 Apr 2024cs.CV cs.AI cs.CLcited by 42

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

There are a thousand ways to caption an image. Contrastive Language Pretraining (CLIP) on the other hand, works by mapping an image and its caption to a single vector -- limiting how well CLIP-like models can represent the diverse ways to describe an image. In this work, we introduce Llip, Latent Language Image Pretraining, which models the diversity of captions that could match an image. Llip's vision encoder outputs a set of visual features that are mixed into a final representation by conditioning on information derived from the text. We show that Llip outperforms non-contextualized baselines like CLIP and SigLIP on a variety of tasks even with large-scale encoders. Llip improves zero-shot classification by an average of 2.9% zero-shot classification benchmarks with a ViT-G/14 encoder. Specifically, Llip attains a zero-shot top-1 accuracy of 83.5% on ImageNet outperforming a similarly sized CLIP by 1.4%. We also demonstrate improvement on zero-shot retrieval on MS-COCO by 6.0%. We provide a comprehensive analysis of the components introduced by the method and demonstrate that Llip leads to richer visual representations.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 63 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

40 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Improved baselines for vision-language pre-training15 May
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
26 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2013Describing Textures in the Wildcited by 2,110
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  8. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  9. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  10. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  11. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  12. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  13. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  14. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  15. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  16. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  17. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  18. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  19. 2023Demystifying CLIP Datacited by 265
  20. 2023Data Filtering Networkscited by 247
  21. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  22. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  23. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  24. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  25. 2020Weakly Supervised Lesion Localization With Probabilistic-CAM Poolingcited by 35
  26. 2023CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a 10,000 Budget; An Extra 4,000 Unlocks 81.8% Accuracycited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 22; 2026: 8.

42 papers.

Sort:

By area:

  1. 2024An Introduction to Vision-Language Modelingcited by 175
  2. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  3. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 47
  4. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  5. 2025An Introduction to Flow Matching and Diffusion Modelscited by 33
  6. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
  7. 2024DOTA: Distributional Test-Time Adaptation of Vision-Language Modelscited by 22
  8. 2024CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generationcited by 21
  9. 2025Bayesian Test-Time Adaptation for Vision-Language Modelscited by 21
  10. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  11. 2024EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretationcited by 17
  12. 2025GOAL: Global-local Object Alignment Learningcited by 17
30 more
  1. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  2. 2025SmartCLIP: Modular Vision-language Alignment with Identification Guaranteescited by 14
  3. 2025On the Value of Cross-Modal Misalignment in Multimodal Representation Learningcited by 9
  4. 2025MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretrainingcited by 5
  5. 2024Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIPcited by 4
  6. 2025FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representationscited by 4
  7. 2025EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognitioncited by 4
  8. 2024HyperCLIP: Adapting Vision-Language models with Hypernetworkscited by 3
  9. 2025MV-CLAM: Multi-View Molecular Interpretation with Cross-Modal Projection via Language Modelcited by 2
  10. 2025FOCAL: Fine-Grained Optimal-Transport-Driven Contrastive Alignment of Language and ECGs with Waveform Enhancementcited by 2
  11. 2025Personalized Image Descriptions from Attention Sequencescited by 2
  12. 2025Learning complete and explainable visual representations from itemized text supervisioncited by 2
  13. 2025Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgettingcited by 1
  14. 2025Text-Driven Causal Representation Learning for Source-Free Domain Generalizationcited by 1
  15. 2025Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Modelscited by 1
  16. 2025Synthetic Captions for Open-Vocabulary Zero-Shot Segmentationcited by 1
  17. 2025Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learningcited by 1
  18. 2025RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classificationcited by 1
  19. 2026Self-Supervised Learning from Structural Invariancecited by 1
  20. 2026Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understandingcited by 1
  21. 2026Language-Instructed Vision Embeddings for Controllable and Generalizable Perceptioncited by 1
  22. 2024Attacks on multimodal modelscited by 0
  23. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Surveycited by 0
  24. 2025Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captionscited by 0
  25. 2025Contrastive vision-language learning with paraphrasing and negationcited by 0
  26. 2026GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretrainingcited by 0
  27. 2026No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Modelscited by 0
  28. 2026FAST-GOAL: Fast and Efficient Global-local Object Alignment Learningcited by 0
  29. 2026MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representationscited by 0
  30. 2026Focus and Supplement: Dual-Enhanced Vision Transformer for Multi-Class Anomaly Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DreamLIP: Language-Image Pre-training with Long Captions
  2. 2025Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Study
  3. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
  4. 2024MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization
  5. 2026CLIP Is Shortsighted: Paying Attention Beyond the First Sentence
  6. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captions
  7. 2023Image Captioners Are Scalable Vision Learners Too
  8. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models