mlsanityData to 14 Oct 2026

Image Recognition with Vision and Language Embeddings of VLMs

arXiv 2509.09311 ↗First posted 11 Sep 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models (VLMs) have enabled strong zero-shot classification through image-text alignment. Yet, their purely visual inference capabilities remain under-explored. In this work, we conduct a comprehensive evaluation of both language-guided and vision-only image classification with a diverse set of dual-encoder VLMs, including both well-established and recent models such as SigLIP 2 and RADIOv2.5. The performance is compared in a standard setup on the ImageNet-1k validation set and its label-corrected variant. The key factors affecting accuracy are analysed, including prompt design, class diversity, the number of neighbours in k-NN, and reference set size. We show that language and vision offer complementary strengths, with some classes favouring textual prompts and others better handled by visual similarity. To exploit this complementarity, we introduce a simple, learning-free fusion method based on per-class precision that improves classification performance. The code is available at: https://github.com/gonikisgo/bmvc2025-vlm-image-recognition.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 27 references that are in the map. First posted on arXiv 11 Sep 2025.

Builds on

24 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models5 Jun
  2. 2024Renovating Names in Open-Vocabulary Segmentation Benchmarks14 Mar
  3. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2023Segment Anythingcited by 8,234
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  6. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  7. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  8. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  9. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  10. 2020Are we done with ImageNet?cited by 333
  11. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  12. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
7 more
  1. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  2. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 89
  3. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 61
  4. 2023ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracycited by 19
  5. 2024Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classificationcited by 19
  6. 2023Learning to Name Classes for Vision and Language Modelscited by 12
  7. 2024Flaws of ImageNet, Computer Vision's Favourite Datasetcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Language as a Label: Zero-Shot Multimodal Classification of Everyday Postures under Data Scarcitycited by 1
  2. 2026Fi-ImageNet-1k: An OOD Benchmark From the Inside of the ImageNet-1k Validation Setcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selection
  2. 2023LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions
  3. 2024Unveiling Encoder-Free Vision-Language Models
  4. 2024Why are Visually-Grounded Language Models Bad at Image Classification?
  5. 2025No Labels Needed: Zero-Shot Image Classification with Collaborative Self-Learning
  6. 2024Rethinking VLMs and LLMs for Image Classification
  7. 2026An overview of 3D Vision-Language Models
  8. 2024Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities