mlsanityData to 7 Oct 2026

Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models

arXiv 2104.08066 ↗First posted 16 Apr 2021cs.CL cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

A method for creating a vision-and-language (V&L) model is to extend a language model through structural modifications and V&L pre-training. Such an extension aims to make a V&L model inherit the capability of natural language understanding (NLU) from the original language model. To see how well this is achieved, we propose to evaluate V&L models using an NLU benchmark (GLUE). We compare five V&L models, including single-stream and dual-stream models, trained with the same pre-training. Dual-stream models, with their higher modality independence achieved by approximately doubling the number of parameters, are expected to preserve the NLU capability better. Our main finding is that the dual-stream scores are not much different than the single-stream scores, contrary to expectation. Further analysis shows that pre-training causes the performance drop in NLU tasks with few exceptions. These results suggest that adopting a single-stream structure and devising the pre-training could be an effective method for improving the maintenance of language knowledge in V&L extensions.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

23 of 38 references.

References · 23

  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2018Neural Network Acceptability Judgmentscited by 952
  12. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
11 more
  1. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  2. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  3. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  4. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  5. 2018RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipescited by 74
  6. 2019The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialoguecited by 46
  7. 2020ManyModalQA: Modality Disambiguation and QA over Diverse Inputscited by 38
  8. 2019A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Contextcited by 35
  9. 2017The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representationscited by 23
  10. 2019MeetUp! A Corpus of Joint Activity Dialogues in a Visual Environmentcited by 19
  11. 2018Like a Baby: Visually Situated Neural Language Acquisitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 7; 2024: 2; 2025: 1; 2026: 1.

18 papers.

Sort:

By area:

  1. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  2. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  3. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  4. 2023Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understandingcited by 19
  5. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learnerscited by 17
  6. 2023Is Multimodal Vision Supervision Beneficial to Language?cited by 6
  7. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
  8. 2022Language with Vision: a Study on Grounded Word and Sentence Embeddingscited by 5
  9. 2021TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoningcited by 4
  10. 2022How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input?cited by 4
  11. 2023A Survey on Image-text Multimodal Modelscited by 4
  12. 2022What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledgecited by 3
6 more
  1. 2024Acquiring Linguistic Knowledge from Multimodal Inputcited by 3
  2. 2024Vision-Language Models Align with Human Neural Representations in Concept Processingcited by 2
  3. 2023Table and Image Generation for Investigating Knowledge of Entities in Pre-trained Vision and Language Modelscited by 1
  4. 2023VLIS: Unimodal Language Models Guide Multimodal Language Generationcited by 1
  5. 2023Controlling for Stereotypes in Multimodal Language Model Evaluationcited by 0
  6. 2026Grounded Concreteness: Human-Like Concreteness Sensitivity in Vision-Language Modelscited by 0