mlsanityData to 14 Oct 2026

End-to-end Knowledge Retrieval with Multi-modal Queries

arXiv 2306.00424 ↗First posted 1 Jun 2023cs.CL cs.CV cs.IRcited by 23

Not on the map. Why?

Abstract (arXiv)

We investigate knowledge retrieval with multi-modal queries, i.e. queries containing information split across image and text inputs, a challenging task that differs from previous work on cross-modal retrieval. We curate a new dataset called ReMuQ for benchmarking progress on this task. ReMuQ requires a system to retrieve knowledge from a large corpus by integrating contents from both text and image queries. We introduce a retriever model ``ReViz'' that can directly process input text and images to retrieve relevant knowledge in an end-to-end fashion without being dependent on intermediate modules such as object detectors or caption generators. We introduce a new pretraining task that is effective for learning knowledge retrieval with multimodal queries and also improves performance on downstream tasks. We demonstrate superior performance in retrieval on two datasets (ReMuQ and OK-VQA) under zero-shot settings as well as further improvements when finetuned on these datasets.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

30 of 52 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
18 more
  1. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  2. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
  3. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  4. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  5. 2015A Dataset for Movie Descriptioncited by 195
  6. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  7. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  8. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 126
  9. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 125
  10. 2016Range Loss for Deep Face Recognition with Long-tailcited by 111
  11. 2021WebQA: Multihop and Multimodal QAcited by 109
  12. 2017Smart Mining for Deep Metric Learningcited by 96
  13. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  14. 2020Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioningcited by 36
  15. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  16. 2021Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 25
  17. 2022Improving Biomedical Information Retrieval with Neural Retrieverscited by 20
  18. 2022A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answeringcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 10; 2026: 3.

23 papers.

Sort:

By area:

  1. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  2. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
  3. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  4. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
  5. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 62
  6. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  7. 2024SciMMIR: Benchmarking Scientific Multi-modal Information Retrievalcited by 23
  8. 2024Generative Multi-Modal Knowledge Retrieval with Large Language Modelscited by 22
  9. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Enginescited by 13
  10. 2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7
  11. 2023May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainabilitycited by 5
  12. 2025FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQAcited by 5
11 more
  1. 2025GENIUS: A Generative Framework for Universal Multimodal Searchcited by 5
  2. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  3. 2025OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignmentcited by 5
  4. 2024MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrievalcited by 3
  5. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  6. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  7. 2026MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answeringcited by 3
  8. 2026Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrievalcited by 2
  9. 2024CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Modelcited by 0
  10. 2025Multimodal Information Retrieval for Open World with Edit Distance Weak Supervisioncited by 0
  11. 2026MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignmentcited by 0