mlsanityData to 14 Oct 2026

Supporting Vision-Language Model Inference with Confounder-pruning Knowledge Prompt

arXiv 2205.11100 ↗First posted 23 May 2022cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models are pre-trained by aligning image-text pairs in a common space to deal with open-set visual concepts. To boost the transferability of the pre-trained models, recent works adopt fixed or learnable prompts, i.e., classification weights are synthesized from natural language describing task-relevant categories, to reduce the gap between tasks in the training and test phases. However, how and what prompts can improve inference performance remains unclear. In this paper, we explicitly clarify the importance of including semantic information in prompts, while existing prompting methods generate prompts without exploring the semantic information of textual labels. Manually constructing prompts with rich semantics requires domain expertise and is extremely time-consuming. To cope with this issue, we propose a semantic-aware prompt learning method, namely CPKP, which retrieves an ontological knowledge graph by treating the textual label as a query to extract task-relevant semantic information. CPKP further introduces a double-tier confounder-pruning procedure to refine the derived semantic information. The graph-tier confounders are gradually identified and phased out, inspired by the principle of Granger causality. The feature-tier confounders are demolished by following the maximum entropy principle in information theory. Empirically, the evaluations demonstrate the effectiveness of CPKP, e.g., with two shots, CPKP outperforms the manual-prompt method by 4.64% and the learnable-prompt method by 1.09% on average, and the superiority of CPKP in domain generalization compared to benchmark approaches. Our implementation is available at https://github.com/Mowenyii/CPKP.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 71 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

43 of 71 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  2. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models16 Oct
  3. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2013Describing Textures in the Wildcited by 2,125
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
27 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  5. 2019Natural Adversarial Examplescited by 1,408
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  10. 2019How Can We Know What Language Models Know?cited by 959
  11. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  12. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  13. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  14. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
  15. 2019KG-BERT: BERT for Knowledge Graph Completioncited by 324
  16. 2019Attention on Attention for Image Captioningcited by 289
  17. 2018Bilinear Attention Networkscited by 280
  18. 2016Image Captioning with Semantic Attentioncited by 260
  19. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  20. 2017Graph Embedding Techniques, Applications, and Performance: A Surveycited by 188
  21. 2021Are Graph Augmentations Necessary? Simple Graph Contrastive Learning for Recommendationcited by 173
  22. 2021Generative Causal Explanations for Graph Neural Networkscited by 124
  23. 2019Knowledge Graph Convolutional Networks for Recommender Systemscited by 121
  24. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 114
  25. 2013A Semantic Matching Energy Function for Learning with Multi-relational Datacited by 61
  26. 2021OntoZSL: Ontology-enhanced Zero-shot Learningcited by 45
  27. 2020Generative Adversarial Zero-Shot Relational Learning for Knowledge Graphscited by 30

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimization of Prompt Learning via Multi-Knowledge Representation for Vision-Language Models
  2. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
  3. 2023Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models
  4. 2025InPK: Infusing Prior Knowledge into Prompt for Vision-Language Models
  5. 2024Learning to Prompt with Text Only Supervision for Vision-Language Models
  6. 2025ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models
  7. 2025AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference with Dynamical Text Guidance
  8. 2025Hierarchical Cross-modal Prompt Learning for Vision-Language Models