mlsanityData to 14 Oct 2026

Leveraging Vision-Language Foundation Models for Fine-Grained Downstream Tasks

arXiv 2307.06795 ↗First posted 13 Jul 2023cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language foundation models such as CLIP have shown impressive zero-shot performance on many tasks and datasets, especially thanks to their free-text inputs. However, they struggle to handle some downstream tasks, such as fine-grained attribute detection and localization. In this paper, we propose a multitask fine-tuning strategy based on a positive/negative prompt formulation to further leverage the capacities of the vision-language foundation models. Using the CLIP architecture as baseline, we show strong improvements on bird fine-grained attribute detection and localization tasks, while also increasing the classification performance on the CUB200-2011 dataset. We provide source code for reproducibility purposes: it is available at https://github.com/FactoDeepLearning/MultitaskVLFM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 33 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

32 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Visual Classification via Description from Large Language Models13 Oct
  2. 2021Decoupling Zero-Shot Semantic Segmentation15 Dec

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
18 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  4. 2021ImageNet-21K Pretraining for the Massescited by 613
  5. 2021Extract Free Dense Labels from CLIPcited by 583
  6. 2021Image Segmentation Using Text and Image Promptscited by 512
  7. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  8. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  9. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  10. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  11. 2016Generating Visual Explanationscited by 158
  12. 2022CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classificationcited by 65
  13. 2022DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognitioncited by 62
  14. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  15. 2022Doubly Right Object Recognition: A Why Prompt for Visual Rationalescited by 29
  16. 2023Xplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosiscited by 23
  17. 2023Fine-grained Visual Classification with High-temperature Refinement and Background Suppressioncited by 22
  18. 2022A Novel Plug-in Module for Fine-Grained Visual Classificationcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners
  2. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  3. 2024Multi-modal Attribute Prompting for Vision-Language Models
  4. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
  5. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
  6. 2024Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model
  7. 2025Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approach
  8. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models