mlsanityData to 7 Oct 2026

Waffling around for Performance: Visual Classification with Random Words and Broad Concepts

arXiv 2306.07282 ↗First posted 12 Jun 2023cs.CV cs.LGcited by 100

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The visual classification performance of vision-language models such as CLIP has been shown to benefit from additional semantic knowledge from large language models (LLMs) such as GPT-3. In particular, averaging over LLM-generated class descriptors, e.g. "waffle, which has a round shape", can notably improve generalization performance. In this work, we critically study this behavior and propose WaffleCLIP, a framework for zero-shot visual classification which simply replaces LLM-generated descriptors with random character and word descriptors. Without querying external models, we achieve comparable performance gains on a large number of visual classification tasks. This allows WaffleCLIP to both serve as a low-cost alternative, as well as a sanity check for any future LLM-based vision-language model extensions. We conduct an extensive experimental study on the impact and shortcomings of additional semantics introduced with LLM-generated descriptors, and showcase how - if available - semantic context is better leveraged by querying LLMs for high-level concepts, which we show can be done to jointly resolve potential class name ambiguities. Code is available here: https://github.com/ExplainableML/WaffleCLIP.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 68 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

55 of 68 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  2. 2022Visual Classification via Description from Large Language Models13 Oct
  3. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct
  4. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  5. 2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model17 Aug
  6. 2022Prompt Distribution Learning6 May
  7. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  8. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  9. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2019Natural Adversarial Examplescited by 1,401
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
34 more
  1. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  2. 2018Do Better ImageNet Models Transfer Better?cited by 560
  3. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  4. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  5. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  6. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  7. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  8. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  9. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  10. 2018Hyperspherical Variational Auto-Encoderscited by 188
  11. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  12. 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
  13. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  14. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  15. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  16. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  17. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  18. 2022I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classificationcited by 54
  19. 2017Fine-graind Image Classification via Combining Vision and Languagecited by 51
  20. 2017Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precisioncited by 46
  21. 2022Unleashing the Power of Visual Prompting At the Pixel Levelcited by 37
  22. 2020Evaluating Robustness to Input Perturbations for Neural Machine Translationcited by 36
  23. 2018Towards Robust Neural Machine Translationcited by 35
  24. 2022Using Language to Extend to Unseen Domainscited by 34
  25. 2022Non-isotropy Regularization for Proxy-based Deep Metric Learningcited by 33
  26. 2022Doubly Right Object Recognition: A Why Prompt for Visual Rationalescited by 29
  27. 2022I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classificationcited by 27
  28. 2017How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?cited by 23
  29. 2020ZEST: Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarizationcited by 22
  30. 2021Large-Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptionscited by 21
  31. 2022Integrating Language Guidance into Vision-based Deep Metric Learningcited by 19
  32. 2022A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learningcited by 19
  33. 2021To Augment or Not to Augment? A Comparative Study on Text Augmentation Techniques for Low-Resource NLPcited by 12
  34. 2021The Curious Layperson: Fine-Grained Image Recognition without Expert Labelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 43; 2025: 33; 2026: 13.

100 papers.

Sort:

By area:

  1. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  2. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  3. 2024VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidancecited by 65
  4. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  5. 2024Learning to Prompt with Text Only Supervision for Vision-Language Modelscited by 37
  6. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
  7. 2024Meta-Prompting for Automating Zero-shot Visual Recognition with LLMscited by 28
  8. 2024Label Propagation for Zero-shot Classification with Vision-Language Modelscited by 28
  9. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  10. 2023DesCo: Learning Object Recognition with Rich Language Descriptionscited by 25
  11. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  12. 2024Any-Shift Prompting for Generalization over Distributionscited by 24
38 more
  1. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  2. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  3. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  4. 2024Evolving Interpretable Visual Classifiers with Large Language Modelscited by 18
  5. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contextscited by 16
  6. 2024DCBM: Data-Efficient Visual Concept Bottleneck Modelscited by 16
  7. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14
  8. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 14
  9. 2023Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classificationcited by 13
  10. 2024Rethinking VLMs and LLMs for Image Classificationcited by 12
  11. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classificationcited by 12
  12. 2023LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptionscited by 11
  13. 2024SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detectioncited by 11
  14. 2025Training-Free Personalization via Retrieval and Reasoning on Fingerprintscited by 10
  15. 2024Efficient Test-Time Prompt Tuning for Vision-Language Modelscited by 9
  16. 2024Training-Free Semantic Segmentation via LLM-Supervisioncited by 8
  17. 2025RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understandingcited by 8
  18. 2023Text Descriptions are Compressive and Invariant Representations for Visual Learningcited by 7
  19. 2024PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneckcited by 7
  20. 2024Tree of Attributes Prompt Learning for Vision-Language Modelscited by 7
  21. 2024SenCLIP: Enhancing zero-shot land-use mapping for Sentinel-2 with ground-level promptingcited by 7
  22. 2025Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognitioncited by 7
  23. 2023Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learningcited by 6
  24. 2024Text-Enhanced Zero-Shot Action Recognition: A training-free approachcited by 6
  25. 2024GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Modelscited by 6
  26. 2024Beyond Accuracy: Ensuring Correct Predictions With Correct Rationalescited by 6
  27. 2024Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIPcited by 6
  28. 2024Does VLM Classification Benefit from LLM Description Semantics?cited by 6
  29. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  30. 2025Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Modelcited by 6
  31. 2023Videoprompter: an ensemble of foundational models for zero-shot video understandingcited by 5
  32. 2023From Categories to Classifiers: Name-Only Continual Learning by Exploring the Webcited by 5
  33. 2024Mind the Gap Between Synthetic and Real: Utilizing Transfer Learning to Probe the Boundaries of Stable Diffusion Generated Datacited by 5
  34. 2024Open Vocabulary Multi-Label Video Classificationcited by 5
  35. 2024CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collectionscited by 5
  36. 2025From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selectioncited by 5
  37. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  38. 2024Visual-Semantic Decomposition and Partial Alignment for Document-based Zero-Shot Learningcited by 4

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM meets Vision-Language Models for Zero-Shot One-Class Classification
  2. 2023Vocabulary-free Image Classification
  3. 2023ChatGPT-Powered Hierarchical Comparisons for Image Classification
  4. 2024Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models
  5. 2022Exploiting Category Names for Few-Shot Classification with Vision-Language Models
  6. 2026Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries
  7. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
  8. 2023HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understanding