# Passage Retrieval for Outside-Knowledge Visual Question Answering

arXiv: https://arxiv.org/abs/2105.03938 · HTML: https://mlsanity.com/paper/2105.03938/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 May 2021.

## Built on by (most cited first, 25 of 25)

- [Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?](https://mlsanity.com/paper/2302.11713/) (2023)
- [Retrieval Augmented Visual Question Answering with Outside Knowledge](https://mlsanity.com/paper/2210.03809/) (2022)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering](https://mlsanity.com/paper/2309.17133/) (2023)
- [Retrieval-Enhanced Machine Learning](https://mlsanity.com/paper/2205.01230/) (2022)
- [PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers](https://mlsanity.com/paper/2402.08327/) (2024)
- [End-to-end Knowledge Retrieval with Multi-modal Queries](https://mlsanity.com/paper/2306.00424/) (2023)
- [Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literature](https://mlsanity.com/paper/2305.11033/) (2023)
- [Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning](https://mlsanity.com/paper/2311.08110/) (2023)
- [LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection](https://mlsanity.com/paper/2207.12888/) (2022)
- [Rethinking Privacy in Machine Learning Pipelines from an Information Flow Control Perspective](https://mlsanity.com/paper/2311.15792/) (2023)
- [A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answering](https://mlsanity.com/paper/2304.13649/) (2023)
- [A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering](https://mlsanity.com/paper/2201.05299/) (2022)
- [Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning](https://mlsanity.com/paper/2205.02355/) (2022)
- [MeMemo: On-device Retrieval Augmentation for Private and Personalized Text Generation](https://mlsanity.com/paper/2407.01972/) (2024)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering](https://mlsanity.com/paper/2306.16478/) (2023)
- [Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering](https://mlsanity.com/paper/2210.10176/) (2022)
- [Multimodal Inverse Cloze Task for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2301.04366/) (2023)
- [From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Model](https://mlsanity.com/paper/2508.00955/) (2025)
- [Enhancing Low-Resource Relation Representations through Multi-View Decoupling](https://mlsanity.com/paper/2312.17267/) (2023)
- [MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval](https://mlsanity.com/paper/2411.08334/) (2024)
- [The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges](https://mlsanity.com/paper/2303.02411/) (2023)
- [Exploring Annotation-free Image Captioning with Retrieval-augmented Pseudo Sentence Generation](https://mlsanity.com/paper/2307.14750/) (2023)
- [CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model](https://mlsanity.com/paper/2411.12287/) (2024)

## Abstract (arXiv)

In this work, we address multi-modal information needs that contain text questions and images by focusing on passage retrieval for outside-knowledge visual question answering. This task requires access to outside knowledge, which in our case we define to be a large unstructured passage collection. We first conduct sparse retrieval with BM25 and study expanding the question with object names and image captions. We verify that visual clues play an important role and captions tend to be more informative than object names in sparse retrieval. We then construct a dual-encoder dense retriever, with the query encoder being LXMERT, a multi-modal pre-trained transformer. We further show that dense retrieval significantly outperforms sparse retrieval that uses object expansion. Moreover, dense retrieval matches the performance of sparse retrieval that leverages human-generated captions.