mlsanityData to 7 Oct 2026

Visual Classification via Description from Large Language Models

arXiv 2210.07183 ↗First posted 13 Oct 2022cs.CV cs.LGcited by 331

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models (VLMs) such as CLIP have shown promising performance on a variety of recognition tasks using the standard zero-shot classification procedure -- computing similarity between the query image and the embedded words for each category. By only using the category name, they neglect to make use of the rich context of additional information that language affords. The procedure gives no intermediate understanding of why a category is chosen, and furthermore provides no mechanism for adjusting the criteria used towards this decision. We present an alternative framework for classification with VLMs, which we call classification by description. We ask VLMs to check for descriptive features rather than broad categories: to find a tiger, look for its stripes; its claws; and more. By basing decisions on these descriptors, we can provide additional cues that encourage using the features we want to be used. In the process, we can get a clear idea of what features the model uses to construct its decision; it gains some level of inherent explainability. We query large language models (e.g., GPT-3) for these descriptors to obtain them in a scalable way. Extensive experiments show our framework has numerous advantages past interpretability. We show improvements in accuracy on ImageNet across distribution shifts; demonstrate the ability to adapt VLMs to recognize concepts unseen during training; and illustrate how descriptors can be edited to effectively mitigate bias compared to the baseline.

Key facts. Cited by 331 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 51 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

31 of 51 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  8. 2016Matching Networks for One Shot Learningcited by 2,246
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  12. 2020In Search of Lost Domain Generalizationcited by 1,042
19 more
  1. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  2. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  3. 2021Robust fine-tuning of zero-shot modelscited by 785
  4. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  6. 2018Do Better ImageNet Models Transfer Better?cited by 560
  7. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  10. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  11. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  12. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  13. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  14. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  15. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  16. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  17. 2020On Robustness and Transferability of Convolutional Neural Networkscited by 98
  18. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  19. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 92; 2024: 119; 2025: 86; 2026: 31.

331 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  3. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  4. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  5. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  6. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  7. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  8. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  9. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
  10. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  11. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  12. 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusioncited by 81
38 more
  1. 2023A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervisioncited by 78
  2. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  3. 2023Fake News in Sheep's Clothing: Robust Fake News Detection Against LLM-Empowered Style Attackscited by 69
  4. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  5. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Modelscited by 67
  6. 2024Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancementcited by 66
  7. 2024The Neglected Tails in Vision-Language Modelscited by 63
  8. 2023Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurtscited by 58
  9. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  10. 2024Language Repository for Long Video Understandingcited by 55
  11. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 55
  12. 2023Multi-Modal Classifiers for Open-Vocabulary Object Detectioncited by 53
  13. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  14. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptionscited by 52
  15. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  16. 2024FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localizationcited by 51
  17. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Promptscited by 48
  18. 2023Prompting Visual-Language Models for Dynamic Facial Expression Recognitioncited by 43
  19. 2023Leveraging Vision-Language Models for Improving Domain Generalization in Image Classificationcited by 41
  20. 2023Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Modelscited by 40
  21. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  22. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptorscited by 39
  23. 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 38
  24. 2024Learning to Prompt with Text Only Supervision for Vision-Language Modelscited by 37
  25. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  26. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
  27. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
  28. 2023Affective Faces for Goal-Driven Dyadic Communicationcited by 34
  29. 2023GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?cited by 33
  30. 2023Revisiting Few-Shot Object Detection with Vision-Language Modelscited by 32
  31. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  32. 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
  33. 2023EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognitioncited by 31
  34. 2024Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Modelscited by 31
  35. 2023A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosiscited by 30
  36. 2024Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discoverycited by 30
  37. 2024Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detectioncited by 30
  38. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30

50 most cited of 331 All 331 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Image Recognition with Vision and Language Embeddings of VLMs
  2. 2024Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities
  3. 2024Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks
  4. 2024Rethinking VLMs and LLMs for Image Classification
  5. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
  6. 2024Does VLM Classification Benefit from LLM Description Semantics?
  7. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions
  8. 2023Vocabulary-free Image Classification