mlsanityData to 7 Oct 2026

Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering

arXiv 2210.10176 ↗First posted 18 Oct 2022cs.CL cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Most Outside-Knowledge Visual Question Answering (OK-VQA) systems employ a two-stage framework that first retrieves external knowledge given the visual question and then predicts the answer based on the retrieved content. However, the retrieved knowledge is often inadequate. Retrievals are frequently too general and fail to cover specific knowledge needed to answer the question. Also, the naturally available supervision (whether the passage contains the correct answer) is weak and does not guarantee question relevancy. To address these issues, we propose an Entity-Focused Retrieval (EnFoRe) model that provides stronger supervision during training and recognizes question-relevant entities to help retrieve more specific knowledge. Experiments show that our EnFoRe model achieves superior retrieval performance on OK-VQA, the currently largest outside-knowledge VQA dataset. We also combine the retrieved knowledge with state-of-the-art VQA models, and achieve a new state-of-the-art performance on OK-VQA.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

39 of 49 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Mask R-CNNcited by 7,259
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
27 more
  1. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  2. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  3. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  4. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  5. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  6. 2018Bilinear Attention Networkscited by 277
  7. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  8. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  9. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  10. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  11. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  12. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  13. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 77
  14. 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
  15. 2019Self-Critical Reasoning for Robust Visual Question Answeringcited by 61
  16. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 59
  17. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  18. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  19. 2021Phrase Retrieval Learns Passage Retrieval, Toocited by 33
  20. 2021Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 25
  21. 2021Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledgecited by 24
  22. 2022A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answeringcited by 13
  23. 2020Improving VQA and its Explanations by Comparing Competing Explanationscited by 12
  24. 2019Generating Question Relevant Captions to Aid Visual Question Answeringcited by 10
  25. 2021Boosted Dense Retrievercited by 10
  26. 2018Attention on Attention: Architectures for Visual Question Answering (VQA)cited by 4
  27. 2018Joint Image Captioning and Question Answeringcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 0; 2025: 4.

7 papers.

  1. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  2. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  3. 2025FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQAcited by 5
  4. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  5. 2023The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challengescited by 2
  6. 2025SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answeringcited by 1
  7. 2025Entity-Augmented Neuroscience Knowledge Retrieval Using Ontology and Semantic Understanding Capability of LLMcited by 0