mlsanityData to 14 Oct 2026

Optimization of Prompt Learning via Multi-Knowledge Representation for Vision-Language Models

arXiv 2404.10357 ↗First posted 16 Apr 2024cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-Language Models (VLMs), such as CLIP, play a foundational role in various cross-modal applications. To fully leverage VLMs' potential in adapting to downstream tasks, context optimization methods like Prompt Tuning are essential. However, one key limitation is the lack of diversity in prompt templates, whether they are hand-crafted or learned through additional modules. This limitation restricts the capabilities of pretrained VLMs and can result in incorrect predictions in downstream tasks. To address this challenge, we propose Context Optimization with Multi-Knowledge Representation (CoKnow), a framework that enhances Prompt Learning for VLMs with rich contextual knowledge. To facilitate CoKnow during inference, we trained lightweight semantic knowledge mappers, which are capable of generating Multi-Knowledge Representation for an input image without requiring additional priors. Experimentally, We conducted extensive experiments on 11 publicly available datasets, demonstrating that CoKnow outperforms a series of previous methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 46 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

33 of 46 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts21 Jul
  2. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  3. 2022Texts as Images in Prompt Tuning for Multi-Label Image Recognition23 Nov
  4. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  5. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  6. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  7. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  8. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  9. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  11. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  12. 2021Robust fine-tuning of zero-shot modelscited by 797
12 more
  1. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  2. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  3. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  4. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  5. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  6. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  7. 2022Context-Aware Robust Fine-Tuningcited by 28
  8. 2023LMEye: An Interactive Perception Network for Large Language Modelscited by 27
  9. 2023Visual and Textual Prior Guided Mask Assemble for Few-Shot Segmentation and Beyondcited by 19
  10. 2016CIFAR-10: KNN-based Ensemble of Classifierscited by 16
  11. 2023CM-MaskSD: Cross-Modality Masked Self-Distillation for Referring Image Segmentationcited by 15
  12. 2022Fine-grained Retrieval Prompt Tuningcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
  2. 2025Hierarchical Cross-modal Prompt Learning for Vision-Language Models
  3. 2025Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Models
  4. 2025InPK: Infusing Prior Knowledge into Prompt for Vision-Language Models
  5. 2023CoPL: Contextual Prompt Learning for Vision-Language Understanding
  6. 2024Learning to Prompt with Text Only Supervision for Vision-Language Models
  7. 2024HPT++: Hierarchically Prompting Vision-Language Models with Multi-Granularity Knowledge Generation and Improved Structure Modeling
  8. 2022Multitask Vision-Language Prompt Tuning