mlsanityData to 7 Oct 2026

Learning Concise and Descriptive Attributes for Visual Recognition

arXiv 2308.03685 ↗First posted 7 Aug 2023cs.CVcited by 73

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent advances in foundation models present new opportunities for interpretable visual recognition -- one can first query Large Language Models (LLMs) to obtain a set of attributes that describe each class, then apply vision-language models to classify images via these attributes. Pioneering work shows that querying thousands of attributes can achieve performance competitive with image features. However, our further investigation on 8 datasets reveals that LLM-generated attributes in a large quantity perform almost the same as random words. This surprising finding suggests that significant noise may be present in these attributes. We hypothesize that there exist subsets of attributes that can maintain the classification performance with much smaller sizes, and propose a novel learning-to-search method to discover those concise sets of attributes. As a result, on the CUB dataset, our method achieves performance close to that of massive LLM-generated attributes (e.g., 10k attributes for CUB), yet using only 32 attributes in total to distinguish 200 bird species. Furthermore, our new paradigm demonstrates several additional benefits: higher interpretability and interactivity for humans, and the ability to summarize knowledge for a recognition task.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 62 references that are in the map. First posted on arXiv 7 Aug 2023.

Builds on

40 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Visual Classification via Description from Large Language Models13 Oct

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
27 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  4. 2013Describing Textures in the Wildcited by 2,110
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2020Concept Bottleneck Modelscited by 1,022
  9. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  11. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  12. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  13. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  14. 2022Post-hoc Concept Bottleneck Modelscited by 289
  15. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  16. 2020Concept Whitening for Interpretable Image Recognitioncited by 225
  17. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  18. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  19. 2018Local Rule-Based Explanations of Black Box Decision Systemscited by 165
  20. 2016Grad-CAM: Why did you say that?cited by 159
  21. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  22. 2021IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammographycited by 72
  23. 2020On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectorscited by 39
  24. 2022Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioningcited by 21
  25. 2021L2C: Describing Visual Differences Needs Semantic Understanding of Individualscited by 8
  26. 2021Comparing interpretability and explainability for feature selectioncited by 6
  27. 2021"Average" Approximates "First Principal Component"? An Empirical Analysis on Representations from Neural Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 30; 2025: 26; 2026: 9.

73 papers.

Sort:

By area:

  1. 2024VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidancecited by 65
  2. 2023Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Modelscited by 40
  3. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  4. 2025Language Guided Concept Bottleneck Models for Interpretable Continual Learningcited by 28
  5. 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Frameworkcited by 27
  6. 2024Democratizing Fine-grained Visual Recognition with Large Language Modelscited by 25
  7. 2023Training-free Zero-shot Composed Image Retrieval with Local Concept Rerankingcited by 23
  8. 2024Explain via Any Concept: Concept Bottleneck Model with Open Vocabulary Conceptscited by 20
  9. 2024Evolving Interpretable Visual Classifiers with Large Language Modelscited by 18
  10. 2024Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classificationcited by 18
  11. 2024Eliminating Information Leakage in Hard Concept Bottleneck Models with Supervised, Hierarchical Concept Learningcited by 17
  12. 2024Pre-trained Vision-Language Models Learn Discoverable Visual Conceptscited by 16
38 more
  1. 2024Scene Graph Generation with Role-Playing Large Language Modelscited by 16
  2. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14
  3. 2023CLIP-QDA: An Explainable Concept Bottleneck Modelcited by 13
  4. 2023LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptionscited by 11
  5. 2024SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detectioncited by 11
  6. 2025HiBug2: Efficient and Interpretable Error Slice Discovery for Comprehensive Model Debuggingcited by 11
  7. 2025Interpretable Generative Models through Post-hoc Concept Bottleneckscited by 11
  8. 2024Aligning Knowledge Concepts to Whole Slide Images for Precise Histopathology Image Analysiscited by 10
  9. 2023Driving through the Concept Gridlock: Unraveling Explainability Bottlenecks in Automated Drivingcited by 9
  10. 2025V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizercited by 9
  11. 2024Training-Free Semantic Segmentation via LLM-Supervisioncited by 8
  12. 2024Benchmarking XAI Explanations with Human-Aligned Evaluationscited by 8
  13. 2025Graph Concept Bottleneck Modelscited by 8
  14. 2024PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneckcited by 7
  15. 2024Evidential Concept Embedding Models: Towards Reliable Concept Explanations for Skin Disease Diagnosiscited by 7
  16. 2024Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effortcited by 7
  17. 2024Tree of Attributes Prompt Learning for Vision-Language Modelscited by 7
  18. 2025Attribute-based Visual Reprogramming for Vision-Language Modelscited by 7
  19. 2024Does VLM Classification Benefit from LLM Description Semantics?cited by 6
  20. 2025Enhancing Interpretable Image Classification Through LLM Agents and Conditional Concept Bottleneck Modelscited by 6
  21. 2025Enhancing Concept Localization in CLIP-based Concept Bottleneck Modelscited by 6
  22. 2025Interpretable and Steerable Concept Bottleneck Sparse Autoencoderscited by 6
  23. 2025Importance Sampling for Multi-Negative Multimodal Direct Preference Optimizationcited by 5
  24. 2025Conditional Representation Learning for Customized Taskscited by 5
  25. 2025Partially Shared Concept Bottleneck Modelscited by 5
  26. 2024Improving Concept Alignment in Vision-Language Concept Bottleneck Modelscited by 4
  27. 2025Locality-aware Concept Bottleneck Modelcited by 4
  28. 2025WISE: Weak-Supervision-Guided Step-by-Step Explanations for Multimodal LLMs in Image Classificationcited by 4
  29. 2026Learning Concept Bottleneck Models from Mechanistic Explanationscited by 4
  30. 2024Explainable Concept Generation through Vision-Language Preference Learning for Understanding Neural Networks' Internal Representationscited by 3
  31. 2025Attribute-formed Class-specific Concept Space: Endowing Language Bottleneck Model with Better Interpretability and Scalabilitycited by 3
  32. 2025Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Modelscited by 3
  33. 2024VLLaVO: Mitigating Visual Gap through LLMscited by 2
  34. 2024Cross-Modal Few-Shot Learning: a Generative Transfer Learning Frameworkcited by 2
  35. 2024Verbalized Representation Learning for Interpretable Few-Shot Generalizationcited by 2
  36. 2024Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Modelscited by 2
  37. 2024Real Classification by Description: Extending CLIP's Limits of Part Attributes Recognitioncited by 2
  38. 2025Generate, Transduct, Adapt: Iterative Transduction with VLMscited by 2

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating Attribute Comprehension in Large Vision-Language Models
  2. 2026Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models
  3. 2024ArtVLM: Attribute Recognition Through Vision-Based Prefix Language Modeling
  4. 2025Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition
  5. 2023Investigating the Role of Attribute Context in Vision-Language Models for Object Recognition and Detection
  6. 2024Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities
  7. 2024Rethinking VLMs and LLMs for Image Classification
  8. 2025Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model