mlsanityData to 7 Oct 2026

A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering

arXiv 2201.05299 ↗First posted 14 Jan 2022cs.CV cs.CL cs.IRcited by 13

Not on the map. Why?

Abstract (arXiv)

Outside-knowledge visual question answering (OK-VQA) requires the agent to comprehend the image, make use of relevant knowledge from the entire web, and digest all the information to answer the question. Most previous works address the problem by first fusing the image and question in the multi-modal space, which is inflexible for further fusion with a vast amount of external knowledge. In this paper, we call for a paradigm shift for the OK-VQA task, which transforms the image into plain text, so that we can enable knowledge passage retrieval, and generative question-answering in the natural language space. This paradigm takes advantage of the sheer volume of gigantic knowledge bases and the richness of pre-trained language models. A Transform-Retrieve-Generate framework (TRiG) framework is proposed, which can be plug-and-played with alternative image-to-text models and textual knowledge bases. Experimental results show that our TRiG framework outperforms all state-of-the-art supervised methods by at least 11.1% absolute margin.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 61 of its 67 references that are in the map. First posted on arXiv 14 Jan 2022.

Builds on

61 of 67 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2019Towards VQA Models That Can Readcited by 1,696
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  6. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  9. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  10. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  11. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  12. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  13. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  14. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  15. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  16. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  17. 2018Bilinear Attention Networkscited by 277
  18. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  19. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  20. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  21. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  22. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  23. 2018Open Domain Question Answering Using Early Fusion of Knowledge Bases and Textcited by 167
  24. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  25. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  26. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  27. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  28. 2020Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answeringcited by 122
  29. 2017Simple and Effective Multi-Paragraph Reading Comprehensioncited by 120
  30. 2016Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?cited by 111
  31. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  32. 2019Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answeringcited by 96
  33. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  34. 2017Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answeringcited by 79
  35. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  36. 2019Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answeringcited by 68
  37. 2020GenericsKB: A Knowledge Base of Generic Statementscited by 51
  38. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 3; 2024: 3; 2025: 3.

13 papers.

Sort:

By area:

  1. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  2. 2023End-to-end Knowledge Retrieval with Multi-modal Queriescited by 23
  3. 2024Generative Multi-Modal Knowledge Retrieval with Large Language Modelscited by 22
  4. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18
  5. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  6. 2023Graph Neural Networks in Vision-Language Image Understanding: A Surveycited by 10
  7. 2022Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 7
  8. 2025FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQAcited by 5
  9. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  10. 2024MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrievalcited by 3
  11. 2022Towards Reasoning-Aware Explainable VQAcited by 2
  12. 2023Open-Set Knowledge-Based Visual Question Answering with Inference Pathscited by 2
1 more
  1. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answeringcited by 2