mlsanityData to 7 Oct 2026

Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding

arXiv 2306.08832 ↗First posted 15 Jun 2023cs.CVcited by 40

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-Language Models (VLMs), such as CLIP, exhibit strong image-text comprehension abilities, facilitating advances in several downstream tasks such as zero-shot image classification, image-text retrieval, and text-to-image generation. However, the compositional reasoning abilities of existing VLMs remains subpar. The root of this limitation lies in the inadequate alignment between the images and captions in the pretraining datasets. Additionally, the current contrastive learning objective fails to focus on fine-grained grounding components like relations, actions, and attributes, resulting in "bag-of-words" representations. We introduce a simple and effective method to improve compositional reasoning in VLMs. Our method better leverages available datasets by refining and expanding the standard image-text contrastive learning framework. Our approach does not require specific annotations and does not incur extra parameters. When integrated with CLIP, our technique yields notable improvement over state-of-the-art baselines across five vision-language compositional benchmarks. We open-source our code at https://github.com/lezhang7/Enhance-FineGrained.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 70 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

53 of 70 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality23 May
  2. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?13 Dec
  3. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Segment Anythingcited by 8,158
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
36 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  9. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  10. 2020Contrastive Learning with Hard Negative Samplescited by 619
  11. 2021Extract Free Dense Labels from CLIPcited by 580
  12. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  13. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  14. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  15. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  16. 2017Sampling Matters in Deep Embedding Learningcited by 285
  17. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  18. 2023Teaching CLIP to Count to Tencited by 194
  19. 2014Learning Fine-grained Image Similarity with Deep Rankingcited by 189
  20. 2017Beyond triplet loss: a deep quadruplet network for person re-identificationcited by 172
  21. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  22. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  23. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  24. 2018Deep Metric Learning with Hierarchical Triplet Losscited by 112
  25. 2017Smart Mining for Deep Metric Learningcited by 96
  26. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  27. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  28. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  29. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  30. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  31. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
  32. 2021Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentationcited by 40
  33. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  34. 2023Going Beyond Nouns With Vision & Language Models Using Synthetic Datacited by 33
  35. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  36. 2019A weakly supervised adaptive triplet loss for deep metric learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 21; 2026: 13.

40 papers.

Sort:

By area:

  1. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
  2. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 44
  3. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  4. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 31
  5. 2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Modelscited by 30
  6. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
  7. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  8. 2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12
  9. 2025VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Modelscited by 11
  10. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 11
  11. 2025Decoupled Global-Local Alignment for Improving Compositional Understandingcited by 9
  12. 2024Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samplescited by 7
28 more
  1. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  2. 2026Recursive Self-Evolving Agents via Held-Out Selectioncited by 6
  3. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captionscited by 5
  4. 2025Evaluating Compositional Generalisation in VLMs and Diffusion Modelscited by 4
  5. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Modelscited by 4
  6. 2025Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learningcited by 3
  7. 2025CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captionscited by 3
  8. 2026Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplificationcited by 3
  9. 2025Do Vision Encoders Truly Explain Object Hallucination?: Mitigating Object Hallucination via Simple Fine-Grained CLIPScorecited by 2
  10. 2025Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Modelscited by 2
  11. 2025MMGeoLM: Hard Negative Contrastive Learning for Fine-Grained Geometric Understanding in Large Multimodal Modelscited by 2
  12. 2025Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptionscited by 2
  13. 2025ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matchingcited by 2
  14. 2026Self-Supervised Learning from Structural Invariancecited by 1
  15. 2024A New Method to Capturing Compositional Knowledge in Linguistic Spacecited by 0
  16. 2025Dynamic Relation Inference via Verb Embeddingscited by 0
  17. 2025Contrastive Learning with Enhanced Abstract Representations using Grouped Loss of Abstract Semantic Supervisioncited by 0
  18. 2025Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMscited by 0
  19. 2026Half-Truths Break Similarity-Based Retrievalcited by 0
  20. 2026No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Modelscited by 0
  21. 2026SynMulti: Synthetic-to-Real Learning for Multimodal Video Understandingcited by 0
  22. 2026Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understandingcited by 0
  23. 2026GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representationscited by 0
  24. 2026What Does the Caption Really Say? Counterfactual Phrase Intervention for Compositional Data Selection in Vision-Language Pretrainingcited by 0
  25. 2026Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionalitycited by 0
  26. 2026Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrievalcited by 0
  27. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysiscited by 0
  28. 2026What Makes Synthetic Hard Negatives Work in Vision-Language Pretraining?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024In-Context Learning Improves Compositional Understanding of Vision-Language Models
  2. 2024Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition
  3. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs
  4. 2024Prompting Large Vision-Language Models for Compositional Reasoning
  5. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models
  6. 2026CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
  7. 2026Inference-Time Structural Reasoning for Compositional Vision-Language Understanding
  8. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives