mlsanityData to 14 Oct 2026

Vocabulary-free Image Classification

arXiv 2306.00917 ↗First posted 1 Jun 2023cs.CVcited by 35

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent advances in large vision-language models have revolutionized the image classification paradigm. Despite showing impressive zero-shot capabilities, a pre-defined set of categories, a.k.a. the vocabulary, is assumed at test time for composing the textual prompts. However, such assumption can be impractical when the semantic context is unknown and evolving. We thus formalize a novel task, termed as Vocabulary-free Image Classification (VIC), where we aim to assign to an input image a class that resides in an unconstrained language-induced semantic space, without the prerequisite of a known vocabulary. VIC is a challenging task as the semantic space is extremely large, containing millions of concepts, with hard-to-discriminate fine-grained categories. In this work, we first empirically verify that representing this semantic space by means of an external vision-language database is the most effective way to obtain semantically relevant content for classifying the image. We then propose Category Search from External Databases (CaSED), a method that exploits a pre-trained vision-language model and an external vision-language database to address VIC in a training-free manner. CaSED first extracts a set of candidate categories from captions retrieved from the database based on their semantic similarity to the image, and then assigns to the image the best matching candidate category according to the same vision-language model. Experiments on benchmark datasets validate that CaSED outperforms other complex vision-language frameworks, while being efficient with much fewer parameters, paving the way for future research in this direction.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 65 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

48 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Improved baselines for vision-language pre-training15 May
  2. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  3. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2017Billion-scale similarity search with GPUscited by 2,497
33 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  2. 2013Describing Textures in the Wildcited by 2,125
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  11. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  13. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  14. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  15. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  16. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 455
  17. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  18. 2020SCAN: Learning to Classify Images without Labelscited by 352
  19. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  20. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  21. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  22. 2019Connecting Vision and Language with Localized Narrativescited by 226
  23. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  24. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  25. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  26. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  27. 2020Automatically Discovering and Learning New Visual Categories with Ranking Statisticscited by 143
  28. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 123
  29. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  30. 2017Self-supervised learning of visual features through embedding images into text topic spacescited by 43
  31. 2020Grafit: Learning fine-grained image representations with coarse labelscited by 43
  32. 2022Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervisioncited by 36
  33. 2023Retrieval-augmented Image Captioningcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 15; 2025: 10; 2026: 6.

35 papers.

Sort:

By area:

  1. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  2. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 89
  3. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  4. 2024Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchycited by 26
  5. 2024Democratizing Fine-grained Visual Recognition with Large Language Modelscited by 25
  6. 2024RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognitioncited by 24
  7. 2023Object Recognition as Next Token Predictioncited by 12
  8. 2025Reasoning Segmentation for Images and Videos: A Surveycited by 12
  9. 2025Taxonomy-Aware Evaluation of Vision-Language Modelscited by 11
  10. 2023Survey on video anomaly detection in dynamic scenes with moving camerascited by 10
  11. 2023Towards Open-Ended Visual Recognition with Large Language Modelcited by 8
  12. 2024Vocabulary-free Image Classification and Semantic Segmentationcited by 8
23 more
  1. 2025On Large Multimodal Models as Open-World Image Classifierscited by 8
  2. 2024Renovating Names in Open-Vocabulary Segmentation Benchmarkscited by 7
  3. 2024Organizing Unstructured Image Collections using Natural Languagecited by 7
  4. 2025[De|Re]constructing VLMs' Reasoning in Countingcited by 6
  5. 2024Vocabulary-Free 3D Instance Segmentation with Vision and Language Assistantcited by 5
  6. 2024TTD: Text-Tag Self-Distillation Enhancing Image-Text Alignment in CLIP to Alleviate Single Tag Biascited by 4
  7. 2024A Hitchhikers Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoningcited by 4
  8. 2024CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Modelscited by 4
  9. 2025Zero-shot Concept Bottleneck Modelscited by 4
  10. 2025Compositional Caching for Training-free Open-vocabulary Attribute Detectioncited by 3
  11. 2025Open Ad-hoc Categorization with Contextualized Feature Learningcited by 3
  12. 2026Are Multimodal Large Language Models Good Annotators for Image Tagging?cited by 3
  13. 2026SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognitioncited by 3
  14. 2024Cross-Modal Few-Shot Learning: a Generative Transfer Learning Frameworkcited by 2
  15. 2025Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMscited by 2
  16. 2024Grounding Descriptions in Images informs Zero-Shot Visual Recognitioncited by 1
  17. 2025Vocabulary-free few-shot learning for Vision-Language Modelscited by 1
  18. 2026Large Multimodal Models as General In-Context Classifierscited by 1
  19. 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Modelscited by 0
  20. 2025Auto-Vocabulary 3D Object Detectioncited by 0
  21. 2026ReCCur: A Recursive Corner-Case Curation Framework for Robust Vision-Language Understanding in Open and Edge Scenarioscited by 0
  22. 2026Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queriescited by 0
  23. 2026Specificity-aware reinforcement learning for fine-grained open-world classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Open-Vocabulary to Vocabulary-Free Semantic Segmentation
  2. 2022Visual Classification via Description from Large Language Models
  3. 2023Towards Open Vocabulary Learning: A Survey
  4. 2022Exploiting Category Names for Few-Shot Classification with Vision-Language Models
  5. 2024Retrieval-enriched zero-shot image classification in low-resource domains
  6. 2025Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model
  7. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentation
  8. 2024Evolving Interpretable Visual Classifiers with Large Language Models