mlsanityData to 7 Oct 2026

Scalable Performance Analysis for Vision-Language Models

arXiv 2305.18786 ↗First posted 30 May 2023cs.CV cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Joint vision-language models have shown great performance over a diverse set of tasks. However, little is known about their limitations, as the high dimensional space learned by these models makes it difficult to identify semantic errors. Recent work has addressed this problem by designing highly controlled probing task benchmarks. Our paper introduces a more scalable solution that relies on already annotated benchmarks. Our method consists of extracting a large set of diverse features from a vision-language benchmark and measuring their correlation with the output of the target model. We confirm previous findings that CLIP behaves like a bag of words model and performs better with nouns and verbs; we also uncover novel insights such as CLIP getting confused by concrete words. Our framework is available at https://github.com/MichiganNLP/Scalable-VLM-Probing and can be used with other multimodal models and benchmarks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 49 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

24 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations1 Jul
  2. 2021What Vision-Language Models `See' when they See Scenes15 Sep

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  11. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  12. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
10 more
  1. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  2. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  3. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  4. 2022Visual Spatial Reasoningcited by 288
  5. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  6. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  7. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  8. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  9. 2020Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressionscited by 33
  10. 2022SEAL : Interactive Tool for Systematic Error Analysis and Labelingcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generationcited by 10
  2. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  2. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  3. 2025Resampling Benchmark for Efficient Comprehensive Evaluation of Large Vision-Language Models
  4. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?
  5. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  6. 2025Test-Time Consistency in Vision Language Models
  7. 2025Evaluating Robustness of Vision-Language Models Under Noisy Conditions
  8. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation