mlsanityData to 7 Oct 2026

What Vision-Language Models `See' when they See Scenes

arXiv 2109.07301 ↗First posted 15 Sep 2021cs.CLcited by 8

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Images can be described in terms of the objects they contain, or in terms of the types of scene or place that they instantiate. In this paper we address to what extent pretrained Vision and Language models can learn to align descriptions of both types with images. We compare 3 state-of-the-art models, VisualBERT, LXMERT and CLIP. We find that (i) V&L models are susceptible to stylistic biases acquired during pretraining; (ii) only CLIP performs consistently well on both object- and scene-level descriptions. A follow-up ablation study shows that CLIP uses object-level information in the visual modality to align with scene-level textual descriptions.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 52 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

34 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr
  2. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels14 Mar

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
20 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  3. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  4. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  5. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  6. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  7. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  8. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  9. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  10. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  11. 2020Experience Grounds Languagecited by 241
  12. 2019Connecting Vision and Language with Localized Narrativescited by 225
  13. 2021Are Convolutional Neural Networks or Transformers more like human vision?cited by 109
  14. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  15. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  16. 2020MiniVLM: A Smaller and Faster Vision-Language Modelcited by 51
  17. 2020GRUEN for Evaluating Linguistic Quality of Generated Textcited by 37
  18. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34
  19. 2020NeuSpell: A Neural Spelling Correction Toolkitcited by 31
  20. 2021Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Casecited by 20

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 3.

8 papers.

  1. 2023Bridging the Digital Divide: Performance Variation across Socio-Economic Factors in Vision-Language Modelscited by 11
  2. 2024Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Costcited by 5
  3. 2023HL Dataset: Visually-grounded Description of Scenes, Actions and Rationalescited by 4
  4. 2024Why context matters in VQA and Reasoning: Semantic interventions for VLM input modalitiescited by 4
  5. 2022Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptionscited by 2
  6. 2023Scalable Performance Analysis for Vision-Language Modelscited by 2
  7. 2024Resilience through Scene Context in Visual Referring Expression Generationcited by 2
  8. 2022Implementing and Experimenting with Diffusion Models for Text-to-Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Object-centric Binding in Contrastive Language-Image Pretraining
  2. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  3. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder
  4. 2026The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding
  5. 2026ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models
  6. 2025A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling
  7. 2026Language-Conditioned Visual Grounding with CLIP Multilingual
  8. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understanding