mlsanityData to 7 Oct 2026

Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering

arXiv 2309.17133 ↗First posted 29 Sep 2023cs.CL cs.CVcited by 51

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Knowledge-based Visual Question Answering (KB-VQA) requires VQA systems to utilize knowledge from external knowledge bases to answer visually-grounded questions. Retrieval-Augmented Visual Question Answering (RA-VQA), a strong framework to tackle KB-VQA, first retrieves related documents with Dense Passage Retrieval (DPR) and then uses them to answer questions. This paper proposes Fine-grained Late-interaction Multi-modal Retrieval (FLMR) which significantly improves knowledge retrieval in RA-VQA. FLMR addresses two major limitations in RA-VQA's retriever: (1) the image representations obtained via image-to-text transforms can be incomplete and inaccurate and (2) relevance scores between queries and documents are computed with one-dimensional embeddings, which can be insensitive to finer-grained relevance. FLMR overcomes these limitations by obtaining image representations that complement those from the image-to-text transforms using a vision model aligned with an existing text-based retriever through a simple alignment network. FLMR also encodes images and questions using multi-dimensional embeddings to capture finer-grained relevance between queries and documents. FLMR significantly improves the original RA-VQA retriever's PRRecall@5 by approximately 8%. Finally, we equipped RA-VQA with two state-of-the-art large multi-modal/language models to achieve ~61% VQA score in the OK-VQA dataset.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 48 of its 56 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

48 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Retrieval Augmented Visual Question Answering with Outside Knowledge7 Oct
  2. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct

Other references · 46

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2017Billion-scale similarity search with GPUscited by 2,477
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
34 more
  1. 2019Towards VQA Models That Can Readcited by 1,696
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  4. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  7. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  10. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  11. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  12. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  13. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  14. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  15. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  16. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  17. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  18. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  19. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  20. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  21. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  22. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  23. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  24. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  25. 2017Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answeringcited by 79
  26. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  27. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  28. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  29. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  30. 2021Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 25
  31. 2019Bilinear Graph Networks for Visual Question Answeringcited by 22
  32. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18
  33. 2022Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 7
  34. 2023FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answeringcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 19; 2026: 20.

51 papers.

Sort:

By area:

  1. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  2. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  3. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
  4. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  5. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Modelscited by 24
  6. 2025Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challengescited by 19
  7. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Countingcited by 17
  8. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  9. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  10. 2025Domain-RAG: Retrieval-Guided Compositional Image Generation for Cross-Domain Few-Shot Object Detectioncited by 14
  11. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  12. 2024Grounding Language Models for Visual Entity Recognitioncited by 12
38 more
  1. 2024MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agentscited by 11
  2. 2026MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chainscited by 11
  3. 2025mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQAcited by 10
  4. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answeringcited by 9
  5. 2024VACoDe: Visual Augmented Contrastive Decodingcited by 7
  6. 2026SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glassescited by 6
  7. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  8. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  9. 2026Taxonomy of the Retrieval System Framework: Pitfalls and Paradigmscited by 4
  10. 2024Q&A Prompts: Discovering Rich Visual Clues through Mining Question-Answer Prompts for VQA requiring Diverse World Knowledgecited by 3
  11. 2024Knowledge Condensation and Reasoning for Knowledge-based VQAcited by 3
  12. 2025AR-RAG: Autoregressive Retrieval Augmentation for Image Generationcited by 3
  13. 2025Learning Event Completeness for Weakly Supervised Video Anomaly Detectioncited by 3
  14. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  15. 2025HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generationcited by 3
  16. 2026Weakly Supervised Video Anomaly Detection with Anomaly-Connected Components and Intention Reasoningcited by 3
  17. 2024Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibitscited by 2
  18. 2025DiN: Diffusion Model for Robust Medical VQA with Semantic Noisy Labelscited by 2
  19. 2025Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answeringcited by 2
  20. 2026QualiRAG: Retrieval-Augmented Generation for Visual Quality Understandingcited by 2
  21. 2026V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrievalcited by 2
  22. 2026CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answeringcited by 2
  23. 2026MINER: Mining Multimodal Internal Representation for Efficient Retrievalcited by 2
  24. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  25. 2025From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answeringcited by 1
  26. 2026LEMUR: Learned Multi-Vector Retrievalcited by 1
  27. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  28. 2025A Knowledge Noise Mitigation Framework for Knowledge-based Visual Question Answeringcited by 0
  29. 2026Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answeringcited by 0
  30. 2026AMES: Approximate Multi-modal Enterprise Search via Late Interaction Retrievalcited by 0
  31. 2026Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMscited by 0
  32. 2026From Clouds to Hallucinations: Atmospheric Retrieval Hijacking in Remote Sensing Vision-Language RAGcited by 0
  33. 2026MicroWorld: Empowering Multimodal Large Language Models to Bridge the Microscopic Domain Gap with Multimodal Attribute Graphcited by 0
  34. 2026Learning Relative Representations for Fine-Grained Multimodal Alignment with Limited Datacited by 0
  35. 2026PROTOCOL: Late Interaction Retrieval for Protein Homolog Searchcited by 0
  36. 2026ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAGcited by 0
  37. 2026Salient Knowledge Pathways: Sparse Cross-Modal Routing for Efficient Knowledge-Intensive Multimodal Question Answeringcited by 0
  38. 2026FashionKG-RAG: Knowledge Graph-Enhanced Retrieval-Augmented Generation for Fashion Question Answeringcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering
  2. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers
  3. 2022A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQA
  4. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  5. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledge
  6. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
  7. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  8. 2024Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answering