mlsanityData to 14 Oct 2026

Passage Retrieval for Outside-Knowledge Visual Question Answering

arXiv 2105.03938 ↗First posted 9 May 2021cs.IRcited by 25

Not on the map. Why?

Abstract (arXiv)

In this work, we address multi-modal information needs that contain text questions and images by focusing on passage retrieval for outside-knowledge visual question answering. This task requires access to outside knowledge, which in our case we define to be a large unstructured passage collection. We first conduct sparse retrieval with BM25 and study expanding the question with object names and image captions. We verify that visual clues play an important role and captions tend to be more informative than object names in sparse retrieval. We then construct a dual-encoder dense retriever, with the query encoder being LXMERT, a multi-modal pre-trained transformer. We further show that dense retrieval significantly outperforms sparse retrieval that uses object expansion. Moreover, dense retrieval matches the performance of sparse retrieval that leverages human-generated captions.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 May 2021.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 12; 2024: 6; 2025: 1.

25 papers.

Sort:

By area:

  1. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  2. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 94
  3. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  4. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  5. 2022Retrieval-Enhanced Machine Learningcited by 47
  6. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  7. 2023End-to-end Knowledge Retrieval with Multi-modal Queriescited by 23
  8. 2023Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literaturecited by 21
  9. 2023Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learningcited by 21
  10. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18
  11. 2023Rethinking Privacy in Machine Learning Pipelines from an Information Flow Control Perspectivecited by 16
  12. 2023A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answeringcited by 14
13 more
  1. 2022A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answeringcited by 13
  2. 2022Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuningcited by 13
  3. 2024MeMemo: On-device Retrieval Augmentation for Private and Personalized Text Generationcited by 13
  4. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  5. 2023Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answeringcited by 10
  6. 2022Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 7
  7. 2023Multimodal Inverse Cloze Task for Knowledge-based Visual Question Answeringcited by 5
  8. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  9. 2023Enhancing Low-Resource Relation Representations through Multi-View Decouplingcited by 3
  10. 2024MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrievalcited by 3
  11. 2023The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challengescited by 2
  12. 2023Exploring Annotation-free Image Captioning with Retrieval-augmented Pseudo Sentence Generationcited by 0
  13. 2024CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Modelcited by 0