mlsanityData to 7 Oct 2026

LOVM: Language-Only Vision Model Selection

arXiv 2306.08893 ↗First posted 15 Jun 2023cs.CV cs.AI cs.LGcited by 12

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Pre-trained multi-modal vision-language models (VLMs) are becoming increasingly popular due to their exceptional performance on downstream vision applications, particularly in the few- and zero-shot settings. However, selecting the best-performing VLM for some downstream applications is non-trivial, as it is dataset and task-dependent. Meanwhile, the exhaustive evaluation of all available VLMs on a novel application is not only time and computationally demanding but also necessitates the collection of a labeled dataset for evaluation. As the number of open-source VLM variants increases, there is a need for an efficient model selection strategy that does not require access to a curated evaluation dataset. This paper proposes a novel task and benchmark for efficiently evaluating VLMs' zero-shot performance on downstream applications without access to the downstream task dataset. Specifically, we introduce a new task LOVM: Language-Only Vision Model Selection, where methods are expected to perform both model selection and performance prediction based solely on a text description of the desired downstream application. We then introduced an extensive LOVM benchmark consisting of ground-truth evaluations of 35 pre-trained VLMs and 23 datasets, where methods are expected to rank the pre-trained VLMs and predict their zero-shot performance.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 73 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

33 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  10. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  11. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  12. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
20 more
  1. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  2. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 155
  3. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  4. 2021Assessing Generalization of SGD via Disagreementcited by 116
  5. 2020Predicting Neural Network Accuracy from Weightscited by 107
  6. 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
  7. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  8. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  9. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  10. 2021Transferability Estimation using Bhattacharyya Class Separabilitycited by 53
  11. 2022PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Taskscited by 30
  12. 2022How stable are Transferability Metrics evaluations?cited by 30
  13. 2017Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approachcited by 27
  14. 2022Towards Better Selective Classificationcited by 21
  15. 2018Efficient Image Dataset Classification Difficulty Estimation for Predicting Deep-Learning Accuracycited by 14
  16. 2019Measuring Dataset Granularitycited by 8
  17. 2021Ranking Models in Unlabeled New Environmentscited by 8
  18. 2022What can we Learn by Predicting Accuracy?cited by 3
  19. 2022Assessing the Value of Transfer Learning Metrics for RF Domain Adaptationcited by 1
  20. 2022Neural Data-to-Text Generation Based on Small Datasets: Comparing the Added Value of Two Semi-Supervised Learning Approaches on Top of a Large Language Modelcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 0; 2026: 3.

12 papers.

  1. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  2. 2023Open World Object Detection in the Era of Foundation Modelscited by 9
  3. 2023Towards Robust Multi-Modal Reasoning via Model Selectioncited by 6
  4. 2024Improving Fine-grained Visual Understanding in VLMs through Text-Only Trainingcited by 5
  5. 2024Improving Medical Diagnostics with Vision-Language Models: Convex Hull-Based Uncertainty Analysiscited by 3
  6. 2023TLDR: Text Based Last-layer Retraining for Debiasing Image Classifierscited by 2
  7. 2024The Inter-Intra Modal Measure: A Predictive Lens on Fine-Tuning Outcomes in Vision-Language Modelscited by 1
  8. 2024Learning to Rank Pre-trained Vision-Language Models for Downstream Taskscited by 1
  9. 2024CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detectioncited by 0
  10. 2026Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queriescited by 0
  11. 2026Model Specific Task Similarity for Vision Language Model Selection via Layer Conductancecited by 0
  12. 2026CODE: Cross-Modal Calibration and Dynamic Suppression for Open World Object Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selection
  2. 2025Vision-Language Model Selection and Reuse for Downstream Adaptation
  3. 2024LM4LV: A Frozen Large Language Model for Low-level Vision Tasks
  4. 2025Mordal: Automated Pretrained Model Selection for Vision Language Models
  5. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation
  6. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  7. 2026Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
  8. 2026An Effective Router for Vision-Language Model Selection