mlsanityData to 7 Oct 2026

Finding Structural Knowledge in Multimodal-BERT

arXiv 2203.09306 ↗First posted 17 Mar 2022cs.CL cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

In this work, we investigate the knowledge learned in the embeddings of multimodal-BERT models. More specifically, we probe their capabilities of storing the grammatical structure of linguistic data and the structure learned over objects in visual data. To reach that goal, we first make the inherent structure of language and visuals explicit by a dependency parse of the sentences that describe the image and by the dependencies between the object regions in the image, respectively. We call this explicit visual structure the textitscene tree, that is based on the dependency tree of the language description. Extensive probing experiments show that the multimodal-BERT models do not encode these scene trees.Code available at urlhttps://github.com/VSJMilewski/multimodal-probes.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 28 of its 42 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

28 of 42 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
16 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  4. 2019Designing and Interpreting Probes with Control Taskscited by 608
  5. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  6. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  7. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  8. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  9. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  10. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  11. 2021Conditional probing: measuring usable information beyond a baselinecited by 61
  12. 2019Visually Grounded Neural Syntax Acquisitioncited by 48
  13. 2020Investigating Gender Bias in BERTcited by 40
  14. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  15. 2020Visually Grounded Compound PCFGscited by 34
  16. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 3; 2024: 1.

7 papers.

  1. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  2. 2022MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Taskscited by 42
  3. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  4. 2022Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignmentcited by 17
  5. 2023Assessment of Pre-Trained Models Across Languages and Grammarscited by 2
  6. 2023Semantic Composition in Visually Grounded Language Modelscited by 1
  7. 2024Towards Knowledge-Infused Automated Disease Diagnosis Assistantcited by 0