mlsanityData to 7 Oct 2026

CV-Probes: Studying the interplay of lexical and world knowledge in visually grounded verb understanding

arXiv 2409.01389 ↗First posted 2 Sep 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

How do vision-language (VL) transformer models ground verb phrases and do they integrate contextual and world knowledge in this process? We introduce the CV-Probes dataset, containing image-caption pairs involving verb phrases that require both social knowledge and visual context to interpret (e.g., "beg"), as well as pairs involving verb phrases that can be grounded based on information directly available in the image (e.g., "sit"). We show that VL models struggle to ground VPs that are strongly context-dependent. Further analysis using explainable AI techniques shows that such models may not pay sufficient attention to the verb token in the captions. Our results suggest a need for improved methodologies in VL model training and evaluation. The code and dataset will be available https://github.com/ivana-13/CV-Probes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 2 Sep 2024.

Builds on

35 of 48 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
23 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  2. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  3. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  4. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  5. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  6. 2020Experience Grounds Languagecited by 241
  7. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  8. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  9. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  10. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  11. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  12. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  13. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  14. 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 61
  15. 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
  16. 2022MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Taskscited by 42
  17. 2020GRUEN for Evaluating Linguistic Quality of Generated Textcited by 37
  18. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34
  19. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  20. 2023A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?cited by 17
  21. 2023Do Language Models' Words Refer?cited by 17
  22. 2023The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Modelscited by 8
  23. 2024Beyond Image-Text Matching: Verb Understanding in Multimodal Transformers Using Guided Maskingcited by 1

Built on by

Not cited yet.