A New Method to Capturing Compositional Knowledge in Linguistic Space
On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP
Abstract (arXiv)
Compositional understanding allows visual language models to interpret complex relationships between objects, attributes, and relations in images and text. However, most existing methods often rely on hard negative examples and fine-tuning, which can overestimate improvements and are limited by the difficulty of obtaining hard negatives. In this work, we introduce Zero-Shot Compositional Understanding (ZS-CU), a novel task that enhances compositional understanding without requiring hard negative training data. We propose YUKINO (Yielded Compositional Understanding Knowledge via Textual Inversion with NO), which uses textual inversion to map unlabeled images to pseudo-tokens in a pre-trained CLIP model. We propose introducing "no" logical regularization to address the issue of token interaction in inversion. Additionally, we suggest using knowledge distillation to reduce the time complexity of textual inversion. Experimental results show that YUKINO outperforms the existing multi-modal SOTA models by over 8% on the SugarCREPE benchmark, and also achieves significant improvements in image retrieval tasks.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 46 references that are in the map. First posted on arXiv 20 Dec 2024.
Builds on
38 of 46 references.
Closest ancestors · 5
Same direction, newest first
- 2024Enhancing Vision-Language Few-Shot Adaptation with Negative Learning19 Mar
- 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality26 Jun
- 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding15 Jun
- 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?13 Dec
- 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct
Other references · 33
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
- 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
21 more
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
- 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
- 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
- 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
- 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
- 2023DataComp: In search of the next generation of multimodal datasetscited by 644
- 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
- 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
- 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
- 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 176
- 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 156
- 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
- 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
- 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
- 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 58
- 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
- 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representationscited by 37
- 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
- 2024Natural Language Inference Improves Compositionality in Vision-Language Modelscited by 8
- 2024Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samplescited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Learning Visual Proxy for Compositional Zero-Shot Learning
- 2025SCOT: Self-Supervised Contrastive Pretraining For Zero-Shot Compositional Retrieval
- 2024Unified Framework for Open-World Compositional Zero-shot Learning
- 2025Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning
- 2025Compositional Image-Text Matching and Retrieval by Grounding Entities
- 2025Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions
- 2023COLA: A Benchmark for Compositional Text-to-image Retrieval
- 2024Compositional Image Retrieval via Instruction-Aware Contrastive Learning