mlsanityData to 7 Oct 2026

Probing Image-Language Transformers for Verb Understanding

arXiv 2106.09141 ↗First posted 16 Jun 2021cs.CL cs.CVcited by 109

Not on the map. Why?

Cited by 43 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning (e.g., visual question answering and image retrieval). We are interested in shedding light on the quality of their pretrained representations -- in particular, if these models can distinguish different types of verbs or if they rely solely on nouns in a given sentence. To do so, we collect a dataset of image-sentence pairs (in English) consisting of 421 verbs that are either visual or commonly found in the pretraining data (i.e., the Conceptual Captions dataset). We use this dataset to evaluate pretrained image-language transformers and find that they fail more in situations that require verb understanding compared to other parts of speech. We also investigate what category of verbs are particularly challenging.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Builds on 24 of its 30 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

24 of 30 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  10. 2018Object Hallucination in Image Captioningcited by 661
  11. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  12. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
12 more
  1. 2016Visual Relationship Detection with Language Priorscited by 333
  2. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  3. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  4. 2015Visual Semantic Role Labelingcited by 188
  5. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  6. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  7. 2018Grounding Visual Explanationscited by 92
  8. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  9. 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 63
  10. 2020Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressionscited by 33
  11. 2020Are we pretraining it right? Digging deeper into visio-linguistic pretrainingcited by 25
  12. 2018Defoiling Foiled Image Captionscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 16; 2023: 30; 2024: 39; 2025: 16; 2026: 5.

109 papers.

Sort:

By area:

  1. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2022Visual Spatial Reasoningcited by 288
  4. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 233
  5. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  6. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  7. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  8. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  9. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  10. 2022CLIP-Event: Connecting Text and Images with Event Structurescited by 86
  11. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  12. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
38 more
  1. 2024Cross-modal Information Flow in Multimodal Large Language Modelscited by 65
  2. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  3. 2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53
  4. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  5. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51
  6. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  7. 2023Towards In-context Scene Understandingcited by 45
  8. 2023MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledgecited by 43
  9. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
  10. 2024Dialect prejudice predicts AI decisions about people's character, employability, and criminalitycited by 35
  11. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  12. 2023Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Trainingcited by 32
  13. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
  14. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 30
  15. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
  16. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 29
  17. 2022Image Retrieval from Contextual Descriptionscited by 26
  18. 2022ComCLIP: Training-Free Compositional Image and Text Matchingcited by 26
  19. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  20. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  21. 2024FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culturecited by 25
  22. 2024Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patchescited by 22
  23. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  24. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 21
  25. 2023SPOT! Revisiting Video-Language Models for Event Understandingcited by 20
  26. 2023Lego: Learning to Disentangle and Invert Personalized Concepts Beyond Object Appearance in Text-to-Image Diffusion Modelscited by 19
  27. 2023Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Modelscited by 18
  28. 2024Iterated Learning Improves Compositionality in Large Vision-Language Modelscited by 17
  29. 2024Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognitioncited by 17
  30. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  31. 2023OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognitioncited by 16
  32. 2024How and where does CLIP process negation?cited by 16
  33. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  34. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
  35. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptionscited by 14
  36. 2023ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representationcited by 13
  37. 2024Can 3D Vision-Language Models Truly Understand Natural Language?cited by 12
  38. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11

50 most cited of 109 All 109 (JSON)