mlsanityData to 7 Oct 2026

VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models

arXiv 2209.06103 ↗First posted 12 Sep 2022cs.CV cs.AI cs.CLcited by 6

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models trained on large, randomly collected data had significant impact in many areas since they appeared. But as they show great performance in various fields, such as image-text-retrieval, their inner workings are still not fully understood. The current work analyses the true zero-shot capabilities of those models. We start from the analysis of the training corpus assessing to what extent (and which of) the test classes are really zero-shot and how this correlates with individual classes performance. We follow up with the analysis of the attribute-based zero-shot learning capabilities of these models, evaluating how well this classical zero-shot notion emerges from large-scale webly supervision. We leverage the recently released LAION400M data corpus as well as the publicly available pretrained models of CLIP, OpenCLIP, and FLAVA, evaluating the attribute-based zero-shot capabilities on CUB and AWA2 benchmarks. Our analysis shows that: (i) most of the classes in popular zero-shot benchmarks are observed (a lot) during pre-training; (ii) zero-shot performance mainly comes out of models' capability of recognizing class labels, whenever they are present in the text, and a significantly lower performing capability of attribute-based zeroshot learning is only observed when class labels are not used; (iii) the number of the attributes used can have a significant effect on performance, and can easily cause a significant performance decrease.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 41 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

34 of 41 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
22 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  4. 2021Robust fine-tuning of zero-shot modelscited by 785
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  7. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  8. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  9. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  10. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  11. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  12. 2015Label-Embedding for Image Classificationcited by 164
  13. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  14. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  15. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  16. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  17. 2021TransZero: Attribute-guided Transformer for Zero-Shot Learningcited by 53
  18. 2022Vision-and-Language Pretrained Models: A Surveycited by 50
  19. 2022MSDN: Mutually Semantic Distillation Network for Zero-Shot Learningcited by 48
  20. 2022WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Modelscited by 20
  21. 2015A Survey of Current Datasets for Vision and Language Researchcited by 13
  22. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Modelscited by 11

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 1.

6 papers.

  1. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  2. 2023Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Modelscited by 10
  3. 2023What Makes ImageNet Look Unlike LAIONcited by 9
  4. 2024Questionable practices in machine learningcited by 3
  5. 2022On the Transferability of Visual Features in Generalized Zero-Shot Learningcited by 0
  6. 2025VideoGEM: Training-free Action Grounding in Videoscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity
  2. 2026Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries
  3. 2026Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models
  4. 2024Enabling Small Models for Zero-Shot Selection and Reuse through Model Label Learning
  5. 2024The Neglected Tails in Vision-Language Models
  6. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learning
  7. 2024BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models
  8. 2023UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding