mlsanityData to 14 Oct 2026

Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity

arXiv 2306.16048 ↗First posted 28 Jun 2023cs.CV cs.AIcited by 14

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

This paper presents novel benchmarks for evaluating vision-language models (VLMs) in zero-shot recognition, focusing on granularity and specificity. Although VLMs excel in tasks like image captioning, they face challenges in open-world settings. Our benchmarks test VLMs' consistency in understanding concepts across semantic granularity levels and their response to varying text specificity. Findings show that VLMs favor moderately fine-grained concepts and struggle with specificity, often misjudging texts that differ from their training data. Extensive evaluations reveal limitations in current VLMs, particularly in distinguishing between correct and subtly incorrect descriptions. While fine-tuning offers some improvements, it doesn't fully address these issues, highlighting the need for VLMs with enhanced generalization capabilities for real-world applications. This study provides insights into VLM limitations and suggests directions for developing more robust models.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 40 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

28 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
15 more
  1. 2020Contrastive Learning with Hard Negative Samplescited by 628
  2. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  3. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  4. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  5. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  6. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  7. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  8. 2019Connecting Vision and Language with Localized Narrativescited by 226
  9. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  10. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 138
  11. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  12. 2022Can Language Understand Depth?cited by 69
  13. 2022Discovering the Hidden Vocabulary of DALLE-2cited by 43
  14. 2021Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neuronscited by 29
  15. 2022Robustness Analysis of Video-Language Models Against Visual and Language Perturbationscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 2.

14 papers.

Sort:

By area:

  1. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 59
  2. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmarkcited by 33
  3. 2024Visually Consistent Hierarchical Image Classificationcited by 16
  4. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14
  5. 2025MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problemscited by 10
  6. 2025CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methodscited by 5
  7. 2025LR0.FM: Low-Res Benchmark and Improving Robustness for Zero-Shot Classification in Foundation Modelscited by 3
  8. 2025SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluationcited by 2
  9. 2025Language as a Label: Zero-Shot Multimodal Classification of Everyday Postures under Data Scarcitycited by 1
  10. 2025Language-Guided Invariance Probing of Vision-Language Modelscited by 1
  11. 2025Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approachcited by 0
  12. 2025Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMscited by 0
2 more
  1. 2026Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detectioncited by 0
  2. 2026Zero-Shot Captioning for Cultural Heritage: Automated Image Analysis of Traditional Indonesian Clothingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
  2. 2023UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding
  3. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selection
  4. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  5. 2024The Neglected Tails in Vision-Language Models
  6. 2026Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries
  7. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  8. 2024Grounding Descriptions in Images informs Zero-Shot Visual Recognition