A Picture May Be Worth a Hundred Words for Visual Question Answering
Not on the map. Why?
Abstract (arXiv)
How far can we go with textual representations for understanding pictures? In image understanding, it is essential to use concise but detailed image representations. Deep visual features extracted by vision models, such as Faster R-CNN, are prevailing used in multiple tasks, and especially in visual question answering (VQA). However, conventional deep visual features may struggle to convey all the details in an image as we humans do. Meanwhile, with recent language models' progress, descriptive text may be an alternative to this problem. This paper delves into the effectiveness of textual representations for image understanding in the specific context of VQA. We propose to take description-question pairs as input, instead of deep visual features, and fed them into a language-only Transformer model, simplifying the process and the computational cost. We also experiment with data augmentation techniques to increase the diversity in the training set and avoid learning statistical bias. Extensive evaluations have shown that textual representations require only about a hundred words to compete with deep visual features on both VQA 2.0 and VQA-CP v2.
Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Builds on 53 of its 65 references that are in the map. First posted on arXiv 25 Jun 2021.
Builds on
53 of 65 references.
References · 53
- 2017Attention Is All You Needcited by 60,106
- 2015Deep Residual Learning for Image Recognitioncited by 51,950
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
- 2014Microsoft COCO: Common Objects in Contextcited by 17,550
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,625
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,713
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
- 2015VQA: Visual Question Answeringcited by 2,495
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,319
38 more
- 2018Deep contextualized word representationscited by 2,286
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,220
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,551
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,487
- 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,452
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
- 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,294
- 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,068
- 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,058
- 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
- 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 802
- 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
- 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
- 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
- 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
- 2018Bilinear Attention Networkscited by 278
- 2015Stacked Attention Networks for Image Question Answeringcited by 273
- 2016Image Captioning with Semantic Attentioncited by 260
- 2019Connecting Vision and Language with Localized Narrativescited by 225
- 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
- 2019RUBi: Reducing Unimodal Biases in Visual Question Answeringcited by 184
- 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
- 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
- 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
- 2017Searching for Activation Functionscited by 162
- 2020In Defense of Grid Features for Visual Question Answeringcited by 155
- 2020Counterfactual Samples Synthesizing for Robust Visual Question Answeringcited by 153
- 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
- 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
- 2019Relation-Aware Graph Attention Network for Visual Question Answeringcited by 119
- 2019Learning by Abstraction: The Neural State Machinecited by 113
- 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
- 2016Boosting Image Captioning with Attributescited by 76
- 2019MUREL: Multimodal Relational Reasoning for Visual Question Answeringcited by 68
- 2020Learning What Makes a Difference from Counterfactual Examples and Gradient Supervisioncited by 66
- 2020MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answeringcited by 63
- 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
- 2018Learning Visual Question Answering by Bootstrapping Hard Attentioncited by 24
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2021: 1; 2022: 0; 2023: 1; 2024: 0; 2025: 0; 2026: 1.
3 papers.