A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models
Not on the map. Why?
Abstract (arXiv)
Key to tasks that require reasoning about natural language in visual contexts is grounding words and phrases to image regions. However, observing this grounding in contemporary models is complex, even if it is generally expected to take place if the task is addressed in a way that is conductive to generalization. We propose a framework to jointly study task performance and phrase grounding, and propose three benchmarks to study the relation between the two. Our results show that contemporary models demonstrate inconsistency between their ability to ground phrases and solve tasks. We show how this can be addressed through brute-force training on ground phrasing annotations, and analyze the dynamics it creates. Code and at available at https://github.com/lil-lab/phrase_grounding.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 65 of its 85 references that are in the map. First posted on arXiv 6 Sep 2023.
Builds on
65 of 85 references.
References · 65
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2020End-to-End Object Detection with Transformerscited by 7,384
- 2017Mask R-CNNcited by 7,259
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2015VQA: Visual Question Answeringcited by 2,489
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
- 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
- 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
- 2021Grounded Language-Image Pre-trainingcited by 1,147
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
- 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
- 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
- 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
- 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
- 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
- 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
- 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
- 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
- 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
- 2018Bilinear Attention Networkscited by 277
- 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
- 2016Segmentation from Natural Language Expressionscited by 257
- 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
- 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
- 2019Connecting Vision and Language with Localized Narrativescited by 225
- 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 139
- 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
- 2019Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Groundedcited by 127
- 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
- 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
- 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
- 2016Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?cited by 111
- 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
- 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
- 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
- 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
- 2017VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentationcited by 50
- 2019Visually Grounded Neural Syntax Acquisitioncited by 48
- 2020More Grounded Image Captioning by Distilling Image-Text Matching Modelcited by 43
50 most cited of 65 All 65 (JSON)
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.