mlsanityData to 14 Oct 2026

GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks

arXiv 2311.01361 ↗First posted 2 Nov 2023cs.CV cs.CLcited by 119

Not on the map. Why?

Cited by 42 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Automatically evaluating vision-language tasks is challenging, especially when it comes to reflecting human judgments due to limitations in accounting for fine-grained details. Although GPT-4V has shown promising results in various multi-modal tasks, leveraging GPT-4V as a generalist evaluator for these tasks has not yet been systematically explored. We comprehensively validate GPT-4V's capabilities for evaluation purposes, addressing tasks ranging from foundational image-to-text and text-to-image synthesis to high-level image-to-image translations and multi-images to text alignment. We employ two evaluation methods, single-answer grading and pairwise comparison, using GPT-4V. Notably, GPT-4V shows promising agreement with humans across various tasks and evaluation methods, demonstrating immense potential for multi-modal LLMs as evaluators. Despite limitations like restricted visual clarity grading and real-world complex reasoning, its ability to provide human-aligned scores enriched with detailed explanations is promising for universal automatic evaluator.

Key facts. Cited by 119 papers in the map (data through 14 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

25 of 36 references.

References · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
13 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  7. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  8. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  9. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  10. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  11. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 87
  12. 2023ImagenHub: Standardizing the evaluation of conditional image generation modelscited by 77
  13. 2022Personalized Showcases: Generating Multi-Modal Explanations for Recommendationscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 48; 2025: 38; 2026: 27.

119 papers.

Sort:

By area:

  1. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  2. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  3. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  4. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  5. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 201
  6. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 156
  7. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 153
  8. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  9. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  10. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  11. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  12. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 129
38 more
  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  2. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 86
  3. 2024Multi-LoRA Composition for Image Generationcited by 80
  4. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  5. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 74
  6. 2025KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Modelscited by 71
  7. 2024Red Teaming Visual Language Modelscited by 65
  8. 2024ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficultycited by 61
  9. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 57
  10. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  11. 2025Text-to-CadQuery: A New Paradigm for CAD Generation with Scalable Large Model Capabilitiescited by 43
  12. 2024Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applicationscited by 31
  13. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 31
  14. 2024Holistic Evaluation for Interleaved Text-and-Image Generationcited by 28
  15. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteriacited by 27
  16. 2024Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filterscited by 25
  17. 2024CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesiscited by 23
  18. 2024Text2CAD: Generating Sequential CAD Models from Beginner-to-Expert Level Text Promptscited by 23
  19. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?cited by 20
  20. 2024Image2Struct: Benchmarking Structure Extraction for Vision-Language Modelscited by 19
  21. 2025Large Language Models for Computer-Aided Design: A Surveycited by 18
  22. 2024METAL: Towards Multilingual Meta-Evaluationcited by 15
  23. 2024SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinementcited by 15
  24. 2025A Unified Agentic Framework for Evaluating Conditional Image Generationcited by 15
  25. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 15
  26. 2024T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Conceptscited by 14
  27. 2025Multimodal LLMs Can Reason about Aesthetics in Zero-Shotcited by 14
  28. 2025Q-Ponder: A Unified Training Pipeline for Reasoning-based Visual Quality Assessmentcited by 14
  29. 2024UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Texturescited by 13
  30. 2024Can MLLMs Perform Text-to-Image In-Context Learning?cited by 13
  31. 2024LangXAI: Integrating Large Vision Models for Generating Textual Explanations to Enhance Explainability in Visual Perception Taskscited by 13
  32. 2026CADSmith: Multi-Agent CAD Generation with Programmatic Geometric Validationcited by 13
  33. 2024ESREAL: Exploiting Semantic Reconstruction to Mitigate Hallucinations in Vision-Language Modelscited by 12
  34. 2025Dual-Process Image Generationcited by 12
  35. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Modelscited by 11
  36. 2023LLM-SQL-Solver: Can LLMs Determine SQL Equivalence?cited by 10
  37. 2025Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluationcited by 10
  38. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 10

50 most cited of 119 All 119 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Lost in Translation: When GPT-4V(ision) Can't See Eye to Eye with Text. A Vision-Language-Consistency Analysis of VLLMs and Beyond
  2. 2024Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency
  3. 2023Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  5. 2023Assessing GPT4-V on Structured Reasoning Tasks
  6. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  7. 2023Enhancing Medical Task Performance in GPT-4V: A Comprehensive Study on Prompt Engineering Strategies
  8. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models