mlsanityData to 7 Oct 2026

Towards Generative Class Prompt Learning for Fine-grained Visual Recognition

arXiv 2409.01835 ↗First posted 3 Sep 2024cs.CV cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Although foundational vision-language models (VLMs) have proven to be very successful for various semantic discrimination tasks, they still struggle to perform faithfully for fine-grained categorization. Moreover, foundational models trained on one domain do not generalize well on a different domain without fine-tuning. We attribute these to the limitations of the VLM's semantic representations and attempt to improve their fine-grained visual awareness using generative modeling. Specifically, we propose two novel methods: Generative Class Prompt Learning (GCPL) and Contrastive Multi-class Prompt Learning (CoMPLe). Utilizing text-to-image diffusion models, GCPL significantly improves the visio-linguistic synergy in class embeddings by conditioning on few-shot exemplars with learnable class prompts. CoMPLe builds on this foundation by introducing a contrastive learning component that encourages inter-class separation during the generative optimization process. Our empirical results demonstrate that such a generative class prompt learning approach substantially outperform existing methods, offering a better alternative to few shot image recognition challenges. The source code will be made available at: https://github.com/soumitri2001/GCPL.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 72 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

59 of 72 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Consistency-guided Prompt Learning for Vision-Language Models1 Jun
  2. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  3. 2022SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models7 Oct
  4. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021Decoupling Zero-Shot Semantic Segmentation15 Dec
  7. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  8. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  5. 2016Matching Networks for One Shot Learningcited by 2,246
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  9. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  10. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  11. 2019A Closer Look at Few-shot Classificationcited by 679
  12. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  13. 2023Universal Guidance for Diffusion Modelscited by 426
  14. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  15. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  16. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 270
  17. 2018Probabilistic Model-Agnostic Meta-Learningcited by 254
  18. 2020Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learningcited by 179
  19. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  20. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 140
  21. 2023A Neural Space-Time Representation for Text-to-Image Personalizationcited by 123
  22. 2020Few-Shot Classification with Feature Map Reconstruction Networkscited by 116
  23. 2021Pre-training without Natural Imagescited by 100
  24. 2019Lung and Colon Cancer Histopathological Image Dataset (LC25000)cited by 86
  25. 2023CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Notcited by 79
  26. 2024LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIPcited by 60
  27. 2023Intriguing properties of generative classifierscited by 52
  28. 2020On Episodes, Prototypical Networks, and Few-shot Learningcited by 46
  29. 2023A Closer Look at Few-shot Classification Againcited by 46
  30. 2022Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentorscited by 34
  31. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  32. 2019PARN: Position-Aware Relation Networks for Few-Shot Learningcited by 27
  33. 2024Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentationcited by 25
  34. 2019Automated Skin Lesion Classification Using Ensemble of Deep Neural Networks in ISIC 2018: Skin Lesion Analysis Towards Melanoma Detection Challengecited by 16
  35. 2022Bi-directional Feature Reconstruction Network for Fine-Grained Few-Shot Image Classificationcited by 12
  36. 2024You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrievalcited by 12
  37. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Modelscited by 12
  38. 2024Data-Free Generalized Zero-Shot Learningcited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025UniCoRN: Latent Diffusion-based Unified Controllable Image Restoration Network across Multiple Degradationscited by 6
  2. 2025NoTeS-Bank: Benchmarking Neural Transcription and Search for Scientific Notes Understandingcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classification
  2. 2024A streamlined Approach to Multimodal Few-Shot Class Incremental Learning for Fine-Grained Datasets
  3. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization
  4. 2025Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learning
  5. 2025InPK: Infusing Prior Knowledge into Prompt for Vision-Language Models
  6. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  7. 2026Structured-Condensed Prompt Tuning in Vision-Language Models for Fine-grained Image Recognition
  8. 2023DPL: Decoupled Prompt Learning for Vision-Language Models