mlsanityData to 7 Oct 2026

Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers

arXiv 2109.04448 ↗First posted 9 Sep 2021cs.CL cs.CVcited by 71

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pretrained vision-and-language BERTs aim to learn representations that combine information from both modalities. We propose a diagnostic method based on cross-modal input ablation to assess the extent to which these models actually integrate cross-modal information. This method involves ablating inputs from one modality, either entirely or selectively based on cross-modal grounding alignments, and evaluating the model prediction performance on the other modality. Model performance is measured by modality-specific tasks that mirror the model pretraining objectives (e.g. masked language modelling for text). Models that have learned to construct cross-modal representations using both modalities are expected to perform worse when inputs are missing from a modality. We find that recently proposed models have much greater relative difficulty predicting text when visual information is ablated, compared to predicting visual object categories when text is ablated, indicating that these models are not symmetrically cross-modal.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

22 of 34 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2019Attention is not Explanationcited by 916
  11. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
10 more
  1. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  2. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  3. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  4. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  5. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  6. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  7. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  8. 2021What Context Features Can Transformer Language Models Use?cited by 49
  9. 2021Measuring and Increasing Context Usage in Context-Aware Machine Translationcited by 48
  10. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 19; 2023: 15; 2024: 14; 2025: 9; 2026: 10.

71 papers.

Sort:

By area:

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  2. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  3. 2022Multimodal Learning with Transformers: A Surveycited by 399
  4. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  5. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  6. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  7. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  8. 2024Cross-modal Information Flow in Multimodal Large Language Modelscited by 65
  9. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  10. 2022MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Taskscited by 42
  11. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39
  12. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
38 more
  1. 2022DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanationscited by 34
  2. 2022On Explaining Multimodal Hateful Meme Detection Modelscited by 29
  3. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
  4. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  5. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 23
  6. 2022Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysiscited by 22
  7. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  8. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 21
  9. 2022Analyzing Modality Robustness in Multimodal Sentiment Analysiscited by 20
  10. 2023Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Bankscited by 20
  11. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19
  12. 2022Bi-VLDoc: Bidirectional Vision-Language Modeling for Visually-Rich Document Understandingcited by 18
  13. 2023Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Modelscited by 18
  14. 2024How and where does CLIP process negation?cited by 16
  15. 2025SPEX: Scaling Feature Interaction Explanations for LLMscited by 12
  16. 2023Towards Balanced Active Learning for Multimodal Classificationcited by 9
  17. 2023Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Braincited by 9
  18. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  19. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  20. 2021What Vision-Language Models `See' when they See Scenescited by 8
  21. 2022Delving Deeper into Cross-lingual Visual Question Answeringcited by 8
  22. 2023Grounded Image Text Matching with Mismatched Relation Reasoningcited by 8
  23. 2025Multi-modal brain encoding models for multi-modal stimulicited by 8
  24. 2022Finding Structural Knowledge in Multimodal-BERTcited by 7
  25. 2023The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Modelscited by 5
  26. 2021TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoningcited by 4
  27. 2022How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input?cited by 4
  28. 2024Why context matters in VQA and Reasoning: Semantic interventions for VLM input modalitiescited by 4
  29. 2025An Empirical Study on How Video-LLMs Answer Video Questionscited by 4
  30. 2025VISaGE: Understanding Visual Generics and Exceptionscited by 4
  31. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Modelscited by 4
  32. 2023CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehensioncited by 3
  33. 2024Acquiring Linguistic Knowledge from Multimodal Inputcited by 3
  34. 2026Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attentioncited by 3
  35. 2022Persuasion Strategies in Advertisementscited by 2
  36. 2022Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptionscited by 2
  37. 2023Generic Attention-model Explainability by Weighted Relevance Accumulationcited by 2
  38. 2024Interpretable Tensor Fusioncited by 2

50 most cited of 71 All 71 (JSON)