mlsanityData to 7 Oct 2026

VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

arXiv 2112.07566 ↗First posted 14 Dec 2021cs.CL cs.CVcited by 134

Not on the map. Why?

Cited by 64 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose VALSE (Vision And Language Structured Evaluation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on specific linguistic phenomena. VALSE offers a suite of six tests covering various linguistic constructs. Solving these requires models to ground linguistic phenomena in the visual modality, allowing more fine-grained evaluations than hitherto possible. We build VALSE using methods that support the construction of valid foils, and report results from evaluating five widely-used V&L models. Our experiments suggest that current models have considerable difficulty addressing most phenomena. Hence, we expect VALSE to serve as an important benchmark to measure future progress of pretrained V&L models from a linguistic perspective, complementing the canonical task-centred V&L evaluations.

Key facts. Cited by 134 papers in the map (data through 7 Oct 2026). Builds on 51 of its 80 references that are in the map. First posted on arXiv 14 Dec 2021.

Builds on

51 of 80 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
38 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  5. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  6. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  7. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  8. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  9. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  10. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  11. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  12. 2016Visual Dialogcited by 377
  13. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  14. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  15. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  16. 2019Certified Robustness to Adversarial Word Substitutionscited by 164
  17. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 150
  18. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  19. 2018Combining Fact Extraction and Verification with Neural Semantic Matching Networkscited by 120
  20. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  21. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  22. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  23. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  24. 2020Language Models are Open Knowledge Graphscited by 95
  25. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  26. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  27. 2020Grounded Situation Recognitioncited by 77
  28. 2020MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answeringcited by 63
  29. 2020Gradient-based Analysis of NLP Models is Manipulablecited by 41
  30. 2020GRUEN for Evaluating Linguistic Quality of Generated Textcited by 37
  31. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34
  32. 2020Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressionscited by 33
  33. 2021Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQAcited by 30
  34. 2019Challenges and Prospects in Vision and Language Researchcited by 24
  35. 2019VIFIDEL: Evaluating the Visual Fidelity of Image Descriptionscited by 21
  36. 2018Visual Referring Expression Recognition: What Do Systems Actually Learn?cited by 20
  37. 2018Beyond task success: A closer look at jointly learning to see, ask, and GuessWhatcited by 15
  38. 2021Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventionscited by 15

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 29; 2024: 33; 2025: 32; 2026: 31.

134 papers.

Sort:

By area:

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2022Visual Spatial Reasoningcited by 288
  4. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  5. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  6. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  7. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  8. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  9. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51
  10. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  11. 2024Improving fine-grained understanding in image-text pre-trainingcited by 44
  12. 2022MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Taskscited by 42
38 more
  1. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 40
  2. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  3. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  4. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 30
  5. 2024Dynamic Negative Guidance of Diffusion Modelscited by 28
  6. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  7. 2025Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIPcited by 26
  8. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  9. 2024FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culturecited by 25
  10. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 23
  11. 2024VisMin: Visual Minimal-Change Understandingcited by 23
  12. 2025FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputscited by 23
  13. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  14. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 21
  15. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  16. 2023Probing Conceptual Understanding of Large Visual-Language Modelscited by 17
  17. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  18. 2024From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Informationcited by 16
  19. 2024FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correctioncited by 16
  20. 2024How and where does CLIP process negation?cited by 16
  21. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14
  22. 2024Is CLIP the main roadblock for fine-grained open-world perception?cited by 13
  23. 2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12
  24. 2023An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metricscited by 11
  25. 2024CLoVe: Encoding Compositional Language in Contrastive Vision-Language Modelscited by 11
  26. 2025VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Modelscited by 11
  27. 2023Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Modelscited by 10
  28. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  29. 2024CELLO: Causal Evaluation of Large Vision-Language Modelscited by 10
  30. 2025MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problemscited by 10
  31. 2023Dealing with Semantic Underspecification in Multimodal NLPcited by 9
  32. 2023Visual Data-Type Understanding does not emerge from Scaling Vision-Language Modelscited by 9
  33. 2024Natural Language Processing RELIES on Linguisticscited by 9
  34. 2025Decoupled Global-Local Alignment for Improving Compositional Understandingcited by 9
  35. 2023Grounded Image Text Matching with Mismatched Relation Reasoningcited by 8
  36. 2023The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Modelscited by 8
  37. 2024CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluationcited by 8
  38. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understandingcited by 8

50 most cited of 134 All 134 (JSON)