mlsanityData to 14 Oct 2026

TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives

arXiv 2411.02545 ↗First posted 4 Nov 2024cs.CV cs.CLcited by 58

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. This makes the nature of the training data a significant factor in the efficacy of CLIP for downstream tasks. However, the lack of compositional diversity in contemporary image-text datasets limits the compositional reasoning ability of CLIP. We show that generating ``hard'' negative captions via in-context learning and synthesizing corresponding negative images with text-to-image generators offers a solution. We introduce a novel contrastive pre-training strategy that leverages these hard negative captions and images in an alternating fashion to train CLIP. We demonstrate that our method, named TripletCLIP, when applied to existing datasets such as CC3M and CC12M, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark on an equal computational budget, as well as improvements in zero-shot image classification and image retrieval. Our code, models, and data are available at: https://tripletclip.github.io

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 55 of its 61 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

55 of 61 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
38 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  2. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  3. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  4. 2023Adversarial Diffusion Distillationcited by 759
  5. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  6. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  7. 2020Contrastive Learning with Hard Negative Samplescited by 628
  8. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  9. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  10. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  11. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillationcited by 279
  12. 2023Demystifying CLIP Datacited by 270
  13. 2023Data Filtering Networkscited by 250
  14. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  15. 2023Improving CLIP Training with Language Rewritescited by 217
  16. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  17. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  18. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  19. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
  20. 2023Hyperbolic Image-Text Representationscited by 136
  21. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  22. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  23. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  24. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  25. 2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 85
  26. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  27. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  28. 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
  29. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  30. 2024R.A.C.E.: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Modelcited by 58
  31. 2023Learning Mask-aware CLIP Representations for Zero-Shot Segmentationcited by 57
  32. 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
  33. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 52
  34. 2023Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Featurescited by 52
  35. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  36. 2024Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negationscited by 46
  37. 2023Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokenscited by 35
  38. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 34; 2026: 22.

58 papers.

Sort:

By area:

  1. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 59
  2. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 31
  3. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  4. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
  5. 2025RefEdit: A Benchmark and Method for Improving Instruction-based Image Editing Model on Referring Expressionscited by 10
  6. 2025Logic Unseen: Revealing the Logical Blindspots of Vision-Language Modelscited by 7
  7. 2025Vision-Language Models as Differentiable Semantic and Spatial Rewards for Text-to-3D Generationcited by 6
  8. 2025Advancing Compositional Awareness in CLIP with Efficient Fine-Tuningcited by 5
  9. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarkscited by 5
  10. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captionscited by 5
  11. 2025HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 5
  12. 2025Common Data Properties Limit Object-Attribute Binding in CLIPcited by 4
38 more
  1. 2024Learning Visual Composition through Improved Semantic Guidancecited by 3
  2. 2025CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captionscited by 3
  3. 2026Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplificationcited by 3
  4. 2025MMGeoLM: Hard Negative Contrastive Learning for Fine-Grained Geometric Understanding in Large Multimodal Modelscited by 2
  5. 2025SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluationcited by 2
  6. 2025Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptionscited by 2
  7. 2025Understanding Hardness of Vision-Language Compositionality from A Token-level Causal Lenscited by 2
  8. 2025ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matchingcited by 2
  9. 2026DEO: Training-Free Direct Embedding Optimization for Negation-Aware Retrievalcited by 2
  10. 2025From No to Know: Taxonomy, Challenges, and Opportunities for Negation Understanding in Multimodal Foundation Modelscited by 1
  11. 2025Can Text-to-Video Generation help Video-Language Alignment?cited by 1
  12. 2025Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updatescited by 1
  13. 2025Adding simple structure at inference improves Vision-Language Compositionalitycited by 1
  14. 2025Unlearning the Noisy Correspondence Makes CLIP More Robustcited by 1
  15. 2025Structure-aware Contrastive Learning for Diagram Understanding of Multimodal Modelscited by 1
  16. 2025Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Modelscited by 1
  17. 2025Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generationcited by 1
  18. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Trainingcited by 1
  19. 2026PolyGen: Fully Synthetic Vision-Language Training via Multi-Generator Ensemblescited by 1
  20. 2026Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understandingcited by 1
  21. 2026Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inferencecited by 1
  22. 2026Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionalitycited by 1
  23. 2024A New Method to Capturing Compositional Knowledge in Linguistic Spacecited by 0
  24. 2025Dynamic Relation Inference via Verb Embeddingscited by 0
  25. 2025Neural Catalog: Scaling Species Recognition with Catalog of Life-Augmented Generationcited by 0
  26. 2025Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMscited by 0
  27. 2025GenView++: Unifying Adaptive Generative Augmentation and Quality-Driven Supervision for Contrastive Representation Learningcited by 0
  28. 2025Harnessing Synthetic Preference Data for Enhancing Temporal Understanding of Video-LLMscited by 0
  29. 2025Contrastive vision-language learning with paraphrasing and negationcited by 0
  30. 2025SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarkingcited by 0
  31. 2026Auto-Comp: An Automated Pipeline for Scalable Compositional Probing of Contrastive Vision-Language Modelscited by 0
  32. 2026Pseudo Contrastive Learning for Diagram Comprehension in Multimodal Modelscited by 0
  33. 2026Half-Truths Break Similarity-Based Retrievalcited by 0
  34. 2026CAPT: Confusion-Aware Prompt Tuning for Reducing Vision-Language Misalignmentcited by 0
  35. 2026PICS: Pairwise Image Compositing with Spatial Interactionscited by 0
  36. 2026HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 0
  37. 2026GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretrainingcited by 0
  38. 2026No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Modelscited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality
  2. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding
  3. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining
  4. 2023Distilling Knowledge from Text-to-Image Generative Models Improves Visio-Linguistic Reasoning in CLIP
  5. 2026CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
  6. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  7. 2025Compositional Image-Text Matching and Retrieval by Grounding Entities
  8. 2023Improved baselines for vision-language pre-training