mlsanityData to 7 Oct 2026

EchoSight: Advancing Visual-Language Models with Wiki Knowledge

arXiv 2407.12735 ↗First posted 17 Jul 2024cs.CVcited by 61

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Knowledge-based Visual Question Answering (KVQA) tasks require answering questions about images using extensive background knowledge. Despite significant advancements, generative models often struggle with these tasks due to the limited integration of external knowledge. In this paper, we introduce EchoSight, a novel multimodal Retrieval-Augmented Generation (RAG) framework that enables large language models (LLMs) to answer visual questions requiring fine-grained encyclopedic knowledge. To strive for high-performing retrieval, EchoSight first searches wiki articles by using visual-only information, subsequently, these candidate articles are further reranked according to their relevance to the combined text-image query. This approach significantly improves the integration of multimodal knowledge, leading to enhanced retrieval outcomes and more accurate VQA responses. Our experimental results on the Encyclopedic VQA and InfoSeek datasets demonstrate that EchoSight establishes new state-of-the-art results in knowledge-based VQA, achieving an accuracy of 41.8% on Encyclopedic VQA and 31.3% on InfoSeek.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 47 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

35 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs23 Apr
  2. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
21 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  3. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  4. 2020Contrastive Learning with Hard Negative Samplescited by 619
  5. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  6. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  7. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  8. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  9. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  10. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  11. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  12. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  13. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  14. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  15. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  16. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  17. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  18. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  19. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  20. 2023End-to-end Knowledge Retrieval with Multi-modal Queriescited by 23
  21. 2024Cross-modal Retrieval for Knowledge-based Visual Question Answeringcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 22; 2026: 35.

61 papers.

Sort:

By area:

  1. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  2. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  3. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  4. 2024Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answeringcited by 20
  5. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQAcited by 19
  6. 2025CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAGcited by 18
  7. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  8. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  9. 2025Towards General Continuous Memory for Vision-Language Modelscited by 14
  10. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  11. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrievalcited by 10
  12. 2025mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQAcited by 10
38 more
  1. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Strongercited by 6
  2. 2025WikiAutoGen: Towards Multi-Modal Wikipedia-Style Article Generationcited by 5
  3. 2025ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrievalcited by 5
  4. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generationcited by 4
  5. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  6. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  7. 2026Do Multimodal RAG Systems Leak Data? A Comprehensive Evaluation of Membership Inference and Image Caption Retrieval Attackscited by 3
  8. 2026Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenescited by 3
  9. 2026Multimodal Adaptive Retrieval Augmented Generation through Internal Representation Learningcited by 3
  10. 2026ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answeringcited by 3
  11. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  12. 2025MMhops-R1: Multimodal Multi-hop Reasoningcited by 2
  13. 2026Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Modelscited by 2
  14. 2026CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answeringcited by 2
  15. 2026Wiki-R1: Incentivizing Multimodal Reasoning for Knowledge-based VQA via Data and Sampling Curriculumcited by 2
  16. 2026Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answeringcited by 2
  17. 2026MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrievalcited by 2
  18. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  19. 2025StaR-KVQA: Structured Reasoning Traces for Implicit-Knowledge Visual Question Answeringcited by 1
  20. 2025Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generationcited by 1
  21. 2026KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answeringcited by 1
  22. 2026When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMscited by 1
  23. 2026REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignmentcited by 1
  24. 2026MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answeringcited by 1
  25. 2026WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognitioncited by 1
  26. 2026MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generationcited by 1
  27. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Modelscited by 1
  28. 2026POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Managementcited by 1
  29. 2026From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAGcited by 1
  30. 2026MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systemscited by 1
  31. 2026Ground Then Rank: Revisiting Knowledge-Based VQA with Training-Free Entity Identificationcited by 1
  32. 2024CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Modelcited by 0
  33. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  34. 2025VaccineRAG: Boosting Multimodal Large Language Models' Immunity to Harmful RAG Samplescited by 0
  35. 2026Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answeringcited by 0
  36. 2026Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Rankingcited by 0
  37. 2026BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answeringcited by 0
  38. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generationcited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  2. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering
  3. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
  4. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts
  5. 2025ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
  6. 2025QA-Dragon: Query-Aware Dynamic RAG System for Knowledge-Intensive Visual Question Answering
  7. 2026KIRA: Knowledge-Intensive Image Retrieval and Reasoning Architecture for Specialized Visual Domains
  8. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering