mlsanityData to 14 Oct 2026

Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories

arXiv 2306.09224 ↗First posted 15 Jun 2023cs.CVcited by 107

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

We propose Encyclopedic-VQA, a large scale visual question answering (VQA) dataset featuring visual questions about detailed properties of fine-grained categories and instances. It contains 221k unique question+answer pairs each matched with (up to) 5 images, resulting in a total of 1M VQA samples. Moreover, our dataset comes with a controlled knowledge base derived from Wikipedia, marking the evidence to support each answer. Empirically, we show that our dataset poses a hard challenge for large vision+language models as they perform poorly on our dataset: PaLI [14] is state-of-the-art on OK-VQA [37], yet it only achieves 13.0% accuracy on our dataset. Moreover, we experimentally show that progress on answering our encyclopedic questions can be achieved by augmenting large models with a mechanism that retrieves relevant information from the knowledge base. An oracle experiment with perfect retrieval achieves 87.0% accuracy on the single-hop portion of our dataset, and an automatic retrieval-augmented prototype yields 48.8%. We believe that our dataset enables future research on retrieval-augmented vision+language models. It is available at https://github.com/google-research/google-research/tree/master/encyclopedic_vqa .

Key facts. Cited by 107 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 48 of its 60 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

48 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory10 Dec

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
35 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  13. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  14. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 309
  15. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  16. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  17. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  18. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 225
  19. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  20. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  21. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  22. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
  23. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  24. 2015Exploring Models and Data for Image Question Answeringcited by 139
  25. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 126
  26. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 125
  27. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  28. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  29. 2022Language Models are General-Purpose Interfacescited by 97
  30. 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
  31. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 67
  32. 2022All You May Need for VQA are Image Captionscited by 65
  33. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 59
  34. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  35. 2015Visual Madlibs: Fill in the blank Image Generation and Question Answeringcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 24; 2025: 30; 2026: 51.

107 papers.

Sort:

By area:

  1. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
  2. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 119
  3. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 111
  4. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  5. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 80
  6. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 62
  7. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 58
  8. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Modelscited by 49
  9. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  10. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 42
  11. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 42
  12. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmarkcited by 33
38 more
  1. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classificationcited by 32
  2. 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Modelscited by 23
  3. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQAcited by 19
  4. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  5. 2025MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Frameworkcited by 18
  6. 2025CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAGcited by 18
  7. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  8. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  9. 2025Towards General Continuous Memory for Vision-Language Modelscited by 15
  10. 2026According to Me: Long-Term Personalized Referential Memory QAcited by 14
  11. 2024Cross-modal Retrieval for Knowledge-based Visual Question Answeringcited by 13
  12. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  13. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  14. 2025mKG-RAG: Leveraging Multimodal Knowledge Graphs in Retrieval-Augmented Generation for Knowledge-intensive VQAcited by 11
  15. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrievalcited by 10
  16. 2026Gemini Embedding 2: A Native Multimodal Embedding Model from Geminicited by 10
  17. 2024Knowledge-Aware Reasoning over Multimodal Semi-structured Tablescited by 9
  18. 2025ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answeringcited by 8
  19. 2026BrowseComp-V3: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agentscited by 8
  20. 2024HAMMR: HierArchical MultiModal React agents for generic VQAcited by 7
  21. 2025ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Groundingcited by 7
  22. 2024Unified Multimodal Interleaved Document Representation for Retrievalcited by 6
  23. 2025Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Rerankingcited by 6
  24. 2024Look Hear: Gaze Prediction for Speech-directed Human Attentioncited by 5
  25. 2024Web-Scale Visual Entity Recognition: An LLM-Driven Data Approachcited by 5
  26. 2026MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledgecited by 5
  27. 2026ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answeringcited by 5
  28. 2023How (not) to ensemble LVLMs for VQAcited by 4
  29. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generationcited by 4
  30. 2024MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrievalcited by 3
  31. 2024VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generationcited by 3
  32. 2025RefVNLI: Towards Scalable Evaluation of Subject-driven Text-to-image Generationcited by 3
  33. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  34. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  35. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  36. 2026Multimodal Adaptive Retrieval Augmented Generation through Internal Representation Learningcited by 3
  37. 2026CogniVerse: Revolutionizing Multi-Modal Retrieval-Augmented Generation with Cognitive Reflection and Geometric Reasoningcited by 3
  38. 2024Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generationcited by 2

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
  2. 2025Exploring Advanced Techniques for Visual Question Answering: A Comprehensive Comparison
  3. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  4. 2024Disentangling Knowledge-based and Visual Reasoning by Question Decomposition in KB-VQA
  5. 2025Visual question answering: from early developments to recent advances -- a survey
  6. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  7. 2025Fine-Grained Knowledge Structuring and Retrieval for Visual Question Answering
  8. 2022Co-VQA : Answering by Interactive Sub Question Sequence