mlsanityData to 7 Oct 2026

Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering

arXiv 2306.16478 ↗First posted 28 Jun 2023cs.IR cs.CL cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

This paper studies a category of visual question answering tasks, in which accessing external knowledge is necessary for answering the questions. This category is called outside-knowledge visual question answering (OK-VQA). A major step in developing OK-VQA systems is to retrieve relevant documents for the given multi-modal query. Current state-of-the-art asymmetric dense retrieval model for this task uses an architecture with a multi-modal query encoder and a uni-modal document encoder. Such an architecture requires a large amount of training data for effective performance. We propose an automatic data generation pipeline for pre-training passage retrieval models for OK-VQA tasks. The proposed approach leads to 26.9% Precision@5 improvements compared to the current state-of-the-art asymmetric architecture. Additionally, the proposed pre-training approach exhibits a good ability in zero-shot retrieval scenarios.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

37 of 56 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2017Billion-scale similarity search with GPUscited by 2,477
25 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  3. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  4. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  5. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  6. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  7. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  8. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  9. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  10. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  11. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  12. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  13. 2020Domain-specific Knowledge Graphs: A surveycited by 68
  14. 2022All You May Need for VQA are Image Captionscited by 65
  15. 2017Neural Ranking Models with Weak Supervisioncited by 61
  16. 2020ManyModalQA: Modality Disambiguation and QA over Diverse Inputscited by 38
  17. 2022Generative Language Models for Paragraph-Level Question Generationcited by 28
  18. 2021Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 25
  19. 2021Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledgecited by 24
  20. 2022A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQAcited by 18
  21. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18
  22. 2021QACE: Asking Questions to Evaluate an Image Captioncited by 15
  23. 2023A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answeringcited by 14
  24. 2017Relevance-based Word Embeddingcited by 12
  25. 2023Multimodal Inverse Cloze Task for Knowledge-based Visual Question Answeringcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6; 2025: 3.

10 papers.

  1. 2024Evaluating Retrieval Quality in Retrieval-Augmented Generationcited by 90
  2. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  3. 2025ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generationcited by 26
  4. 2024Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Modelscited by 23
  5. 2024Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Modelscited by 20
  6. 2024Learning to Rank for Multiple Retrieval-Augmented Models through Iterative Utility Maximizationcited by 14
  7. 2024Accelerating Retrieval-Augmented Generationcited by 14
  8. 2023Solution for SMART-101 Challenge of ICCV Multi-modal Algorithmic Reasoning Task 2023cited by 1
  9. 2025Plan-and-Refine: Diverse and Comprehensive Retrieval-Augmented Generationcited by 0
  10. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0