mlsanityData to 14 Oct 2026

Real Classification by Description: Extending CLIP's Limits of Part Attributes Recognition

arXiv 2412.13947 ↗First posted 18 Dec 2024cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

In this study, we define and tackle zero shot "real" classification by description, a novel task that evaluates the ability of Vision-Language Models (VLMs) like CLIP to classify objects based solely on descriptive attributes, excluding object class names. This approach highlights the current limitations of VLMs in understanding intricate object descriptions, pushing these models beyond mere object recognition. To facilitate this exploration, we introduce a new challenge and release description data for six popular fine-grained benchmarks, which omit object names to encourage genuine zero-shot learning within the research community. Additionally, we propose a method to enhance CLIP's attribute detection capabilities through targeted training using ImageNet21k's diverse object categories, paired with rich attribute descriptions generated by large language models. Furthermore, we introduce a modified CLIP architecture that leverages multiple resolutions to improve the detection of fine-grained part attributes. Through these efforts, we broaden the understanding of part-attribute recognition in CLIP, improving its performance in fine-grained classification tasks across six popular benchmarks, as well as in the PACO dataset, a widely used benchmark for object-attribute recognition. Code is available at: https://github.com/ethanbar11/grounding_ge_public.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 28 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

20 of 28 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models27 Nov
  2. 2023Learning Concise and Descriptive Attributes for Visual Recognition7 Aug
  3. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Concepts12 Jun
  4. 2022Visual Classification via Description from Large Language Models13 Oct
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  4. 2021ImageNet-21K Pretraining for the Massescited by 613
  5. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  6. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 288
  7. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  8. 2014Efficient Object Localization Using Convolutional Networkscited by 227
  9. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  10. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 151
  11. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  12. 2024When Do We Not Need Larger Vision Models?cited by 67
3 more
  1. 2023Multi-Modal Classifiers for Open-Vocabulary Object Detectioncited by 53
  2. 2023OvarNet: Towards Open-vocabulary Object Attribute Recognitioncited by 43
  3. 2023Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentationcited by 1
  2. 2025LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentationcited by 1
  3. 2026Semantic Capability Acquisition and Specialization During Vision-Language Model Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts
  2. 2026Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models
  3. 2026Explaining CLIP Zero-shot Predictions Through Concepts
  4. 2023Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification
  5. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
  6. 2023AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
  7. 2024Evolving Interpretable Visual Classifiers with Large Language Models
  8. 2024Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class