mlsanityData to 14 Oct 2026

An Evaluation of GPT-4V and Gemini in Online VQA

arXiv 2312.10637 ↗First posted 17 Dec 2023cs.CV cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

While there is much excitement about the potential of large multimodal models (LMM), a comprehensive evaluation is critical to establish their true capabilities and limitations. In support of this aim, we evaluate two state-of-the-art LMMs, GPT-4V and Gemini, on a new visual question answering dataset sourced from an authentic online question answering community. We conduct fine-grained analysis by generating seven types of metadata for nearly 2,000 visual questions, such as image type and the required image processing capabilities. Our zero-shot performance analysis highlights the types of questions that are most challenging for both models, including questions related to "puzzling" topic, with "Identification" user intention, with "Sheet Music" image type, or labeled as "hard" by GPT-4.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 14 references that are in the map. First posted on arXiv 17 Dec 2023.

Builds on

12 of 14 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 11

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  5. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  7. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 96
  8. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 80
  9. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  10. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answeringcited by 23
  11. 2023Fully Authentic Visual Question Answering Dataset from Online Communitiescited by 9

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3.

5 papers.

  1. 2024LOVA3: Learning to Visual Question Answering, Asking and Assessmentcited by 21
  2. 2024Long-Form Answers to Visual Questions from Blind and Low Vision Peoplecited by 18
  3. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 16
  4. 2023Gemini Pro Defeated by GPT-4V: Evidence from Educationcited by 12
  5. 2024BBA: Bi-Modal Behavioral Alignment for Reasoning with Large Vision-Language Modelscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
  2. 2024Realizing Visual Question Answering for Education: GPT-4V as a Multimodal AI
  3. 2025Exploring Advanced Techniques for Visual Question Answering: A Comprehensive Comparison
  4. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V
  5. 2024A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
  6. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
  7. 2025Visual question answering: from early developments to recent advances -- a survey
  8. 2024Towards Flexible Evaluation for Generative Visual Question Answering