Data to 7 Oct 2026

VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations

arXiv 2207.00221 ↗First posted 1 Jul 2022cs.CV cs.CL cs.LGcited by 113

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Pretraining (VLP) models have recently successfully facilitated many cross-modal downstream tasks. Most existing works evaluated their systems by comparing the fine-tuned downstream task performance. However, only average downstream task accuracy provides little information about the pros and cons of each VLP method, let alone provides insights on how the community can improve the systems in the future. Inspired by the CheckList for testing natural language processing, we exploit VL-CheckList, a novel framework to understand the capabilities of VLP models. The proposed method divides the image-texting ability of a VLP model into three categories: objects, attributes, and relations, and uses a novel taxonomy to further break down these three aspects. We conduct comprehensive studies to analyze seven recently popular VLP models via the proposed framework. Results confirm the effectiveness of the proposed method by revealing fine-grained differences among the compared models that were not visible from downstream task-only evaluation. Further results show promising research direction in building better VLP models. Our data and code are available at: https://github.com/om-ai-lab/VL-CheckList.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 40 references that are in the map. First posted on arXiv 1 Jul 2022.

Builds on

34 of 40 references.

Closest ancestors · 1

Same direction, newest first

2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb

Other references · 33

2017Attention Is All You Needcited by 59,932
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
2015A large annotated corpus for learning natural language inferencecited by 1,837
2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548

21 more

2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
2021Dynabench: Rethinking Benchmarking in NLPcited by 447
2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
2020TIDE: A General Toolbox for Identifying Object Detection Errorscited by 113
2021Probing Image-Language Transformers for Verb Understandingcited by 109
2021Learning to Predict Visual Attributes in the Wildcited by 95
2020Grounded Situation Recognitioncited by 77
2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 63
2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39
2019HAKE: Human Activity Knowledge Enginecited by 32
2021VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-wordscited by 14
2022Vision Checklist: Towards Testable Error Analysis of Image Models to Help System Designers Interrogate Model Capabilitiescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 32; 2024: 27; 2025: 26; 2026: 23.

113 papers.

Sort:

By area:

2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
2024An Introduction to Vision-Language Modelingcited by 175
2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
2024Compositional Entailment Learning for Hyperbolic Vision-Language Modelscited by 72
2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 59
2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51

38 more

2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
2023MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledgecited by 43
2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 40
2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
2023Going Beyond Nouns With Vision & Language Models Using Synthetic Datacited by 33
2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 30
2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 30
2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
2022ComCLIP: Training-Free Compositional Image and Text Matchingcited by 26
2023Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Miningcited by 25
2024Towards Multimodal In-Context Learning for Vision & Language Modelscited by 24
2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
2022ConStruct-VL: Data-Free Continual Structured VL Concepts Learningcited by 20
2023SPOT! Revisiting Video-Language Models for Event Understandingcited by 20
2023GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detectioncited by 20
2025Is CLIP ideal? No. Can we fix it? Yes!cited by 20
2024TULIP: Token-length Upgraded CLIPcited by 18
2023Probing Conceptual Understanding of Large Visual-Language Modelscited by 17
2024Iterated Learning Improves Compositionality in Large Vision-Language Modelscited by 17
2024FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correctioncited by 16
2023How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detectioncited by 15
2023Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learnerscited by 14
2023Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMscited by 14
2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
2022FETA: Towards Specializing Foundation Models for Expert Task Applicationscited by 13
2023ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representationcited by 13
2024Can 3D Vision-Language Models Truly Understand Natural Language?cited by 12
2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12
2025CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representationcited by 12
2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
2024CLoVe: Encoding Compositional Language in Contrastive Vision-Language Modelscited by 11
2025VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Modelscited by 11
2023Text-guided Eyeglasses Manipulation with Spatial Constraintscited by 10

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way