mlsanityData to 7 Oct 2026

Cross-modal Retrieval for Knowledge-based Visual Question Answering

arXiv 2401.05736 ↗First posted 11 Jan 2024cs.CL cs.IRcited by 13

Not on the map. Why?

Abstract (arXiv)

Knowledge-based Visual Question Answering about Named Entities is a challenging task that requires retrieving information from a multimodal Knowledge Base. Named entities have diverse visual representations and are therefore difficult to recognize. We argue that cross-modal retrieval may help bridge the semantic gap between an entity and its depictions, and is foremost complementary with mono-modal retrieval. We provide empirical evidence through experiments with a multimodal dual encoder, namely CLIP, on the recent ViQuAE, InfoSeek, and Encyclopedic-VQA datasets. Additionally, we study three different strategies to fine-tune such a model: mono-modal, cross-modal, or joint training. Our method, which combines mono-and cross-modal retrieval, is competitive with billion-parameter models on the three datasets, while being conceptually simpler and computationally cheaper.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 40 of its 60 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

40 of 60 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
28 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2021Transformers in Vision: A Surveycited by 1,009
  3. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  4. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  5. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  6. 2014Memory Networkscited by 354
  7. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  8. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  9. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  10. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  11. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  12. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  13. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  14. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  15. 2019Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answeringcited by 96
  16. 2016Image-embodied Knowledge Representation Learningcited by 68
  17. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  18. 2022Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrievalcited by 57
  19. 2022Retrieval-Enhanced Machine Learningcited by 46
  20. 2018Embedding Multimodal Relational Data for Knowledge Base Completioncited by 34
  21. 2020VisualSem: A High-quality Knowledge Graph for Vision and Languagecited by 30
  22. 2021Embedding Arithmetic of Multimodal Queries for Image Retrievalcited by 21
  23. 2022Hypergraph Transformer: Weakly-supervised Multi-hop Reasoning for Knowledge-based Visual Question Answeringcited by 15
  24. 2022Visual Named Entity Linking: A New Dataset and A Baselinecited by 15
  25. 2020End-to-End Entity Classification on Multimodal Knowledge Graphscited by 12
  26. 2021Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injectioncited by 8
  27. 2023AVIS: Autonomous Visual Information Seeking with Large Language Model Agentcited by 8
  28. 2023Multimodal Inverse Cloze Task for Knowledge-based Visual Question Answeringcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 7.

13 papers.

Sort:

By area:

  1. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 79
  2. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 61
  3. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  4. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  5. 2024Unified Multimodal Interleaved Document Representation for Retrievalcited by 6
  6. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  7. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Modelscited by 3
  8. 2026CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answeringcited by 2
  9. 2026MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answeringcited by 1
  10. 2026Very Efficient Listwise Multimodal Reranking for Long Documentscited by 1
  11. 2026MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systemscited by 1
  12. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generationcited by 0
1 more
  1. 2026MMAgent-R2: Learning to Rerank and Reject for Agentic mRAGcited by 0