mlsanityData to 7 Oct 2026

A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering

arXiv 2311.07536 ↗First posted 13 Nov 2023cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

The emergence of multimodal large models (MLMs) has significantly advanced the field of visual understanding, offering remarkable capabilities in the realm of visual question answering (VQA). Yet, the true challenge lies in the domain of knowledge-intensive VQA tasks, which necessitate not just recognition of visual elements, but also a deep comprehension of the visual information in conjunction with a vast repository of learned knowledge. To uncover such capabilities of MLMs, particularly the newly introduced GPT-4V and Gemini, we provide an in-depth evaluation from three perspectives: 1) Commonsense Knowledge, which assesses how well models can understand visual cues and connect to general knowledge; 2) Fine-grained World Knowledge, which tests the model's skill in reasoning out specific knowledge from images, showcasing their proficiency across various specialized fields; 3) Comprehensive Knowledge with Decision-making Rationales, which examines model's capability to provide logical explanations for its inference, facilitating a deeper analysis from the interpretability perspective. Additionally, we utilize a visual knowledge-enhanced training strategy and multimodal retrieval-augmented generation approach to enhance MLMs, highlighting the future need for advancements in this research direction. Extensive experiments indicate that: a) GPT-4V demonstrates enhanced explanation generation when using composite images as few-shots; b) GPT-4V and other MLMs produce severe hallucinations when dealing with world knowledge; c) Visual knowledge enhanced training and prompting technicals present potential to improve performance. Codes: https://github.com/HITsz-TMG/Cognitive-Visual-Language-Mapper

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 29 of its 30 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

29 of 30 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
17 more
  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  4. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  5. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  6. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  7. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  8. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  9. 2022Linearly Mapping from Image to Text Spacecited by 139
  10. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  11. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  12. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  13. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  14. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  15. 2023LMEye: An Interactive Perception Network for Large Language Modelscited by 27
  16. 2023A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Cluescited by 11
  17. 2023A Neural Divide-and-Conquer Reasoning Framework for Image Retrieval from Linguistically Complex Textcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 14; 2025: 3; 2026: 2.

23 papers.

Sort:

By area:

  1. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  2. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
  3. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  4. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  5. 2023DrugAssist: A Large Language Model for Molecule Optimizationcited by 57
  6. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  7. 2024Comp4D: LLM-Guided Compositional 4D Scene Generationcited by 50
  8. 2024VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Contextcited by 29
  9. 2024Do Vision-Language Models Really Understand Visual Language?cited by 29
  10. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 29
  11. 2024ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Modelscited by 19
  12. 2023Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Modelscited by 18
11 more
  1. 2024PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulationcited by 17
  2. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 15
  3. 2024Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMscited by 6
  4. 2024INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurancecited by 6
  5. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Strongercited by 6
  6. 2023An Evaluation of GPT-4V and Gemini in Online VQAcited by 5
  7. 2024Explore the Hallucination on Low-level Perception for MLLMscited by 3
  8. 2024An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imagingcited by 1
  9. 2026PolyReal: A Benchmark for Real-World Polymer Science Workflowscited by 1
  10. 2025SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answeringcited by 0
  11. 2026Multi-modal Reasoning with LLMs for Visual Semantic Arithmeticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  2. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering
  3. 2024Realizing Visual Question Answering for Education: GPT-4V as a Multimodal AI
  4. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  5. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
  6. 2024Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models
  7. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
  8. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey