Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions
Not on the map. Why?
Abstract (arXiv)
Image captioning models tend to describe images in an object-centric way, emphasising visible objects. But image descriptions can also abstract away from objects and describe the type of scene depicted. In this paper, we explore the potential of a state-of-the-art Vision and Language model, VinVL, to caption images at the scene level using (1) a novel dataset which pairs images with both object-centric and scene descriptions. Through (2) an in-depth analysis of the effect of the fine-tuning, we show (3) that a small amount of curated data suffices to generate scene descriptions without losing the capability to identify object-level concepts in the scene; the model acquires a more holistic view of the image compared to when object-centric descriptions are generated. We discuss the parallels between these results and insights from computational and cognitive science research on scene perception.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 62 references that are in the map. First posted on arXiv 9 Nov 2022.
Builds on
33 of 62 references.
References · 33
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
- 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
- 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
21 more
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
- 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
- 2020Quantifying Attention Flow in Transformerscited by 759
- 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
- 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
- 2018nocaps: novel object captioning at scalecited by 444
- 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
- 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
- 2019Connecting Vision and Language with Localized Narrativescited by 225
- 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
- 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
- 2021Probing Image-Language Transformers for Verb Understandingcited by 109
- 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
- 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
- 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
- 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
- 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 63
- 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 61
- 2021Perceptual Score: What Data Modalities Does Your Model Perceive?cited by 35
- 2021What Vision-Language Models `See' when they See Scenescited by 8
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.