mlsanityData to 14 Oct 2026

An Early Evaluation of GPT-4V(ision)

arXiv 2310.16534 ↗First posted 25 Oct 2023cs.CL cs.CVcited by 31

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we evaluate different abilities of GPT-4V including visual understanding, language understanding, visual puzzle solving, and understanding of other modalities such as depth, thermal, video, and audio. To estimate GPT-4V's performance, we manually construct 656 test instances and carefully evaluate the results of GPT-4V. The highlights of our findings are as follows: (1) GPT-4V exhibits impressive performance on English visual-centric benchmarks but fails to recognize simple Chinese texts in the images; (2) GPT-4V shows inconsistent refusal behavior when answering questions related to sensitive traits such as gender, race, and age; (3) GPT-4V obtains worse results than GPT-4 (API) on language understanding tasks including general language understanding benchmarks and visual commonsense knowledge evaluation benchmarks; (4) Few-shot prompting can improve GPT-4V's performance on both visual understanding and language understanding; (5) GPT-4V struggles to find the nuances between two similar images and solve the easy math picture puzzles; (6) GPT-4V shows non-trivial performance on the tasks of similar modalities to image, such as video and thermal. Our experimental results reveal the ability and limitations of GPT-4V and we hope our paper can provide some insights into the application and research of GPT-4V.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 11 of its 32 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

11 of 32 references.

References · 11

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  7. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  8. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  9. 2018nocaps: novel object captioning at scalecited by 444
  10. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 84
  11. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 17; 2025: 6.

31 papers.

Sort:

By area:

  1. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  2. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  3. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  4. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 90
  5. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  6. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 57
  7. 2023Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMscited by 34
  8. 2025Introducing Visual Perception Token into Multimodal Large Language Modelcited by 26
  9. 2023GPT-4V-AD: Exploring Grounding Potential of VQA-oriented GPT-4V for Zero-shot Anomaly Detectioncited by 23
  10. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 23
  11. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configurationcited by 23
  12. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 18
19 more
  1. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  2. 2024From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesiscited by 17
  3. 2023NERIF: GPT-4V for Automatic Scoring of Drawn Modelscited by 13
  4. 2024Progress and Opportunities of Foundation Models in Bioinformaticscited by 13
  5. 2024LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Groundingcited by 7
  6. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrationscited by 7
  7. 2023GPT-4V(ision) as A Social Media Analysis Enginecited by 6
  8. 20253DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4ocited by 6
  9. 2024Harnessing GPT-4V(ision) for Insurance: A Preliminary Explorationcited by 5
  10. 2024Realizing Visual Question Answering for Education: GPT-4V as a Multimodal AIcited by 5
  11. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Modelscited by 5
  12. 2024BBA: Bi-Modal Behavioral Alignment for Reasoning with Large Vision-Language Modelscited by 4
  13. 2024Developing ChatGPT for Biology and Medicine: A Complete Review of Biomedical Question Answeringcited by 3
  14. 2025A Multimodal, Multilingual, and Multidimensional Pipeline for Fine-grained Crowdsourcing Earthquake Damage Evaluationcited by 3
  15. 2024CollagePrompt: A Benchmark for Budget-Friendly Visual Recognition with GPT-4Vcited by 2
  16. 2024An Eye for an AI: Evaluating GPT-4o's Visual Perception Skills and Geometric Reasoning Skills Using Computer Graphics Questionscited by 2
  17. 2024GPT-4V Explorations: Mining Autonomous Drivingcited by 1
  18. 2023DeepArt: A Benchmark to Advance Fidelity Research in AI-Generated Contentcited by 0
  19. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Modelscited by 0