# Prompting Large Pre-trained Vision-Language Models For Compositional Concept Learning

arXiv: https://arxiv.org/abs/2211.05077 · HTML: https://mlsanity.com/paper/2211.05077/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 9 Nov 2022.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Task-Driven Modular Networks for Zero-Shot Compositional Learning](https://mlsanity.com/paper/1905.05908/) (2019)
- [Learning Graph Embeddings for Compositional Zero-shot Learning](https://mlsanity.com/paper/2102.01987/) (2021)
- [Open World Compositional Zero-Shot Learning](https://mlsanity.com/paper/2101.12609/) (2021)
- [Symmetry and Group in Attribute-Object Compositions](https://mlsanity.com/paper/2004.00587/) (2020)
- [Learning to Compose Soft Prompts for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2204.03574/) (2022)
- [Attributes as Operators: Factorizing Unseen Attribute-Object Compositions](https://mlsanity.com/paper/1803.09851/) (2018)

## Built on by (most cited first, 12 of 12)

- [Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task](https://mlsanity.com/paper/2310.09336/) (2023)
- [Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2303.15230/) (2023)
- [Prompting Language-Informed Distribution for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2305.14428/) (2023)
- [GIPCOL: Graph-Injected Soft Prompting for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2311.05729/) (2023)
- [Learning Clustering-based Prototypes for Compositional Zero-shot Learning](https://mlsanity.com/paper/2502.06501/) (2025)
- [Compositional Zero-shot Learning via Progressive Language-based Observations](https://mlsanity.com/paper/2311.14749/) (2023)
- [TsCA: On the Semantic Consistency Alignment via Conditional Transport for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2408.08703/) (2024)
- [Swing-by Dynamics in Concept Learning and Compositional Generalization](https://mlsanity.com/paper/2410.08309/) (2024)
- [Learning Visual Proxy for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2501.13859/) (2025)
- [Semantically Guided Dynamic Visual Prototype Refinement for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2501.07114/) (2025)
- [EVA: Mixture-of-Experts Semantic Variant Alignment for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2506.20986/) (2025)
- [Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds](https://mlsanity.com/paper/2606.19941/) (2026)

## Abstract (arXiv)

This work explores the zero-shot compositional learning ability of large pre-trained vision-language models(VLMs) within the prompt-based learning framework and propose a model (textitPromptCompVL) to solve the compositonal zero-shot learning (CZSL) problem. textitPromptCompVL makes two design choices: first, it uses a soft-prompting instead of hard-prompting to inject learnable parameters to reprogram VLMs for compositional learning. Second, to address the compositional challenge, it uses the soft-embedding layer to learn primitive concepts in different combinations. By combining both soft-embedding and soft-prompting, textitPromptCompVL achieves state-of-the-art performance on the MIT-States dataset. Furthermore, our proposed model achieves consistent improvement compared to other CLIP-based methods which shows the effectiveness of the proposed prompting strategies for CZSL.