mlsanityData to 7 Oct 2026

GIPCOL: Graph-Injected Soft Prompting for Compositional Zero-Shot Learning

arXiv 2311.05729 ↗First posted 9 Nov 2023cs.CVcited by 16

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Pre-trained vision-language models (VLMs) have achieved promising success in many fields, especially with prompt learning paradigm. In this work, we propose GIP-COL (Graph-Injected Soft Prompting for COmpositional Learning) to better explore the compositional zero-shot learning (CZSL) ability of VLMs within the prompt-based learning framework. The soft prompt in GIPCOL is structured and consists of the prefix learnable vectors, attribute label and object label. In addition, the attribute and object labels in the soft prompt are designated as nodes in a compositional graph. The compositional graph is constructed based on the compositional structure of the objects and attributes extracted from the training data and consequently feeds the updated concept representation into the soft prompt to capture this compositional structure for a better prompting for CZSL. With the new prompting strategy, GIPCOL achieves state-of-the-art AUC results on all three CZSL benchmarks, including MIT-States, UT-Zappos, and C-GQA datasets in both closed and open settings compared to previous non-CLIP as well as CLIP-based methods. We analyze when and why GIPCOL operates well given the CLIP backbone and its training data limitations, and our findings shed light on designing more effective prompts for CZSL

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 36 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

29 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Prompting Large Pre-trained Vision-Language Models For Compositional Concept Learning9 Nov
  2. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learning7 Apr
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models16 Oct

Other references · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2017Graph Attention Networkscited by 8,854
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  11. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  12. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
13 more
  1. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  2. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  3. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  4. 2017Feature Generating Networks for Zero-Shot Learningcited by 257
  5. 2016An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wildcited by 146
  6. 2019Task-Driven Modular Networks for Zero-Shot Compositional Learningcited by 121
  7. 2021Learning Graph Embeddings for Compositional Zero-shot Learningcited by 112
  8. 2021Open World Compositional Zero-Shot Learningcited by 100
  9. 2020Symmetry and Group in Attribute-Object Compositionscited by 89
  10. 2020Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resourcescited by 83
  11. 2022Siamese Contrastive Embedding Network for Compositional Zero-Shot Learningcited by 55
  12. 2021Learning Graph Embeddings for Open World Compositional Zero-Shot Learningcited by 49
  13. 2022KG-SP: Knowledge Guided Simple Primitives for Open World Compositional Zero-Shot Learningcited by 41

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6; 2025: 9.

16 papers.

Sort:

By area:

  1. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  2. 2025Learning Clustering-based Prototypes for Compositional Zero-shot Learningcited by 13
  3. 2023Compositional Zero-shot Learning via Progressive Language-based Observationscited by 12
  4. 2024TsCA: On the Semantic Consistency Alignment via Conditional Transport for Compositional Zero-Shot Learningcited by 6
  5. 2024On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Surveycited by 5
  6. 2024Exploring Prompt Engineering: A Systematic Review with SWOT Analysiscited by 5
  7. 2024SHINE: Saliency-aware HIerarchical NEgative Ranking for Compositional Temporal Groundingcited by 3
  8. 2025Learning Visual Proxy for Compositional Zero-Shot Learningcited by 2
  9. 2025Visual Adaptive Prompting for Compositional Zero-Shot Learningcited by 2
  10. 2024MAC: A Benchmark for Multiple Attributes Compositional Zero-Shot Learningcited by 1
  11. 2025Learning Primitive Relations for Compositional Zero-Shot Learningcited by 1
  12. 2025MSCI: Addressing CLIP's Inherent Limitations for Compositional Zero-Shot Learningcited by 1
4 more
  1. 2025H2em: Learning Hierarchical Hyperbolic Embeddings for Compositional Zero-Shot Learningcited by 1
  2. 2025Semantically Guided Dynamic Visual Prototype Refinement for Compositional Zero-Shot Learningcited by 0
  3. 2025EVA: Mixture-of-Experts Semantic Variant Alignment for Compositional Zero-Shot Learningcited by 0
  4. 2025Proactive Disentangled Modeling of Trigger-Object Pairings for Backdoor Defensecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Decomposed Soft Prompt Guided Fusion Enhancing for Compositional Zero-Shot Learning
  2. 2023CAILA: Concept-Aware Intra-Layer Adapters for Compositional Zero-Shot Learning
  3. 2025Separated Inter/Intra-Modal Fusion Prompts for Compositional Zero-Shot Learning
  4. 2023Prompting Language-Informed Distribution for Compositional Zero-Shot Learning
  5. 2023DRPT: Disentangled and Recurrent Prompt Tuning for Compositional Zero-Shot Learning
  6. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation
  7. 2026Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognition
  8. 2023Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning