mlsanityData to 14 Oct 2026

Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts

arXiv 2307.11661 ↗First posted 21 Jul 2023cs.CV cs.AI cs.CLcited by 50

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Contrastive pretrained large Vision-Language Models (VLMs) like CLIP have revolutionized visual representation learning by providing good performance on downstream datasets. VLMs are 0-shot adapted to a downstream dataset by designing prompts that are relevant to the dataset. Such prompt engineering makes use of domain expertise and a validation dataset. Meanwhile, recent developments in generative pretrained models like GPT-4 mean they can be used as advanced internet search tools. They can also be manipulated to provide visual information in any structure. In this work, we show that GPT-4 can be used to generate text that is visually descriptive and how this can be used to adapt CLIP to downstream tasks. We show considerable improvements in 0-shot transfer accuracy on specialized fine-grained datasets like EuroSAT (~7%), DTD (~7%), SUN397 (~4.6%), and CUB (~3.3%) when compared to CLIP's default prompt. We also design a simple few-shot adapter that learns to choose the best possible sentences to construct generalizable classifiers that outperform the recently proposed CoCoOP by ~2% on average and by over 4% on 4 specialized fine-grained datasets. The code, prompts, and auxiliary text dataset is available at https://github.com/mayug/VDT-Adapter.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 58 references that are in the map. First posted on arXiv 21 Jul 2023.

Builds on

37 of 58 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  2. 2022Visual Classification via Description from Large Language Models13 Oct
  3. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2013Describing Textures in the Wildcited by 2,125
19 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  3. 2019How Can We Know What Language Models Know?cited by 959
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  5. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  6. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  7. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  8. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  9. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 255
  10. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  11. 2018Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoderscited by 206
  12. 2015Label-Embedding for Image Classificationcited by 165
  13. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  14. 2016Learning Visual N-Grams from Web Datacited by 95
  15. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 89
  16. 2023KoLA: Carefully Benchmarking World Knowledge of Large Language Modelscited by 85
  17. 2015Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptionscited by 64
  18. 2022I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classificationcited by 54
  19. 2022BaseTransformers: Attention over base data-points for One Shot Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 28; 2025: 12; 2026: 4.

50 papers.

Sort:

By area:

  1. 2024Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancementcited by 67
  2. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptionscited by 54
  3. 2024FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localizationcited by 51
  4. 2024Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detectioncited by 32
  5. 2024Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentationcited by 25
  6. 2023Zero-Shot Robustification of Zero-Shot Modelscited by 24
  7. 2024Scene Graph Generation with Role-Playing Large Language Modelscited by 16
  8. 2024SAM2CLIP2SAM: Vision Language Model for Segmentation of 3D CT Scans for Covid-19 Detectioncited by 14
  9. 2023Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Imagescited by 13
  10. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  11. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classificationcited by 13
  12. 2024Multi-modal Attribute Prompting for Vision-Language Modelscited by 12
38 more
  1. 2024Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignmentcited by 12
  2. 2023LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptionscited by 11
  3. 2023Generating Action-conditioned Prompts for Open-vocabulary Video Action Recognitioncited by 8
  4. 2024Enhancing Vision-Language Few-Shot Adaptation with Negative Learningcited by 8
  5. 2024Tree of Attributes Prompt Learning for Vision-Language Modelscited by 7
  6. 2025AquaticCLIP: A Vision-Language Foundation Model for Underwater Scene Analysiscited by 7
  7. 2025EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Modelscited by 6
  8. 2024Test-Time Multimodal Backdoor Detection by Contrastive Promptingcited by 5
  9. 2024Visual-Semantic Decomposition and Partial Alignment for Document-based Zero-Shot Learningcited by 4
  10. 2025Attribute-formed Class-specific Concept Space: Endowing Language Bottleneck Model with Better Interpretability and Scalabilitycited by 4
  11. 2025HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learningcited by 3
  12. 2024Exploiting LMM-based knowledge for image classification taskscited by 2
  13. 2024Proactive Schemes: A Survey of Adversarial Attacks for Social Goodcited by 2
  14. 2024Enhancing Visual Classification using Comparative Descriptorscited by 2
  15. 2024Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Modelscited by 2
  16. 2025Generate, Transduct, Adapt: Iterative Transduction with VLMscited by 2
  17. 2025MADS: Multi-Attribute Document Supervision for Zero-Shot Image Classificationcited by 2
  18. 2025Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Modelscited by 2
  19. 2025Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehiclescited by 2
  20. 2026Dual-Modality Anchor-Guided Filtering for Test-time Prompt Tuningcited by 2
  21. 2023Representing visual classification as a linear combination of wordscited by 1
  22. 2023GELDA: A generative language annotation framework to reveal visual biases in datasetscited by 1
  23. 2024Transparency Attacks: How Imperceptible Image Layers Can Fool AI Perceptioncited by 1
  24. 2024Exploring the Distinctiveness and Fidelity of the Descriptions Generated by Large Vision-Language Modelscited by 1
  25. 2024CoAPT: Context Attribute words for Prompt Tuningcited by 1
  26. 2024MCGMark: An Encodable and Robust Online Watermark for Tracing LLM-Generated Malicious Codecited by 1
  27. 2024AFANet: Adaptive Frequency-Aware Network for Weakly-Supervised Few-Shot Semantic Segmentationcited by 1
  28. 2026Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapterscited by 1
  29. 2024Video Annotator: A framework for efficiently building video classifiers using vision-language models and active learningcited by 0
  30. 2024Exploiting Alpha Transparency In Language And Vision-Based AI Systemscited by 0
  31. 2024Optimization of Prompt Learning via Multi-Knowledge Representation for Vision-Language Modelscited by 0
  32. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?cited by 0
  33. 2024LMM-Regularized CLIP Embeddings for Image Classificationcited by 0
  34. 2025Advancing Vision-based Human Action Recognition: Exploring Vision-Language CLIP Model for Generalisation in Domain-Independent Taskscited by 0
  35. 2025Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representationscited by 0
  36. 2025Visual Species Recognition with Large Multimodal Models as Post-Hoc Correctorscited by 0
  37. 2026Decompose and Transfer: CoT-Prompting Enhanced Alignment for Open-Vocabulary Temporal Action Detectioncited by 0
  38. 2026Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners
  2. 2024CLIPArTT: Adaptation of CLIP to New Domains at Test Time
  3. 2024Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
  4. 2025Generalizable Prompt Learning of CLIP: A Brief Overview
  5. 2024Learning to Prompt with Text Only Supervision for Vision-Language Models
  6. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  7. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder
  8. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMs