mlsanityData to 7 Oct 2026

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

arXiv 2302.11713 ↗First posted 23 Feb 2023cs.CV cs.AI cs.CLcited by 203

Not on the map. Why?

Cited by 94 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-trained vision and language models have demonstrated state-of-the-art capabilities over existing tasks involving images and texts, including visual question answering. However, it remains unclear whether these models possess the capability to answer questions that are not only querying visual content but knowledge-intensive and information-seeking. In this study, we introduce InfoSeek, a visual question answering dataset tailored for information-seeking questions that cannot be answered with only common sense knowledge. Using InfoSeek, we analyze various pre-trained visual question answering models and gain insights into their characteristics. Our findings reveal that state-of-the-art pre-trained multi-modal models (e.g., PaLI-X, BLIP2, etc.) face challenges in answering visual information-seeking questions, but fine-tuning on the InfoSeek dataset elicits models to use fine-grained knowledge that was learned during their pre-training. Furthermore, we show that accurate visual entity recognition can be used to improve performance on InfoSeek by retrieving relevant documents, showing a significant space for improvement.

Key facts. Cited by 203 papers in the map (data through 7 Oct 2026). Builds on 52 of its 88 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

52 of 88 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  12. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  13. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  14. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  15. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  16. 2021ImageNet-21K Pretraining for the Massescited by 607
  17. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  18. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  19. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  20. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  21. 2023Augmented Language Models: a Surveycited by 398
  22. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  23. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  24. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  25. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  26. 2022UL2: Unifying Language Learning Paradigmscited by 292
  27. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  28. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  29. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  30. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
  31. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  32. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  33. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  34. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  35. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  36. 2021FRUIT: Faithfully Reflecting Updated Information in Textcited by 39
  37. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  38. 2021Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 25

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 47; 2025: 54; 2026: 93.

203 papers.

Sort:

By area:

  1. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  2. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  3. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  4. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  5. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  6. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
  7. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  8. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  9. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  10. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  11. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  12. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 79
38 more
  1. 2024Safety of Multimodal Large Language Models on Images and Textscited by 70
  2. 2024Instruct-Imagen: Image Generation with Multi-modal Instructioncited by 65
  3. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
  4. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  5. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 61
  6. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  7. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  8. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  9. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  10. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  11. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 41
  12. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  13. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  14. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Imagescited by 36
  15. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  16. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classificationcited by 31
  17. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 31
  18. 2024DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentationcited by 30
  19. 2026Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Modelscited by 30
  20. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Modelscited by 28
  21. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  22. 2024MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editingcited by 25
  23. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Modelscited by 24
  24. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answeringcited by 23
  25. 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Modelscited by 23
  26. 2024LOVA3: Learning to Visual Question Answering, Asking and Assessmentcited by 21
  27. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  28. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledgecited by 20
  29. 2024Granular Privacy Control for Geolocation with Vision Language Modelscited by 19
  30. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQAcited by 19
  31. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  32. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  33. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 18
  34. 2025CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAGcited by 18
  35. 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 18
  36. 2026OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agentscited by 18
  37. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 17
  38. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17

50 most cited of 203 All 203 (JSON)