mlsanityData to 7 Oct 2026

Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation

arXiv 2401.06591 ↗First posted 12 Jan 2024cs.CLcited by 123

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Assessing long-form responses generated by Vision-Language Models (VLMs) is challenging. It not only requires checking whether the VLM follows the given instruction but also verifying whether the text output is properly grounded on the given image. Inspired by the recent approach of evaluating LMs with LMs, in this work, we propose to evaluate VLMs with VLMs. For this purpose, we present a new feedback dataset called the Perception Collection, encompassing 15K customized score rubrics that users might care about during assessment. Using the Perception Collection, we train Prometheus-Vision, the first open-source VLM evaluator model that can understand the user-defined score criteria during evaluation. Prometheus-Vision shows the highest Pearson correlation with human evaluators and GPT-4V among open-source models, showing its effectiveness for transparent and accessible evaluation of VLMs. We open-source our code, dataset, and model at https://github.com/kaistAI/prometheus-vision

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 48 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

25 of 48 references.

References · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  11. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  12. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
13 more
  1. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  2. 2022Self-critiquing models for assisting human evaluatorscited by 372
  3. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  4. 2022Large Language Models Are Reasoning Teacherscited by 318
  5. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  6. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  7. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  8. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  9. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  10. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  11. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 54
  12. 2023Improving Automatic VQA Evaluation Using Large Language Modelscited by 37
  13. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteriacited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 43; 2026: 63.

123 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  3. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  4. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  5. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  6. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  7. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  8. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  9. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  10. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  11. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  12. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 40
38 more
  1. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  2. 2025CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoningcited by 28
  3. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  4. 2024VisionArena: 230K Real World User-VLM Conversations with Preference Labelscited by 22
  5. 2025Personalized Preference Fine-tuning of Diffusion Modelscited by 22
  6. 2026AutoFigure: Generating and Refining Publication-Ready Scientific Illustrationscited by 20
  7. 2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
  8. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  9. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 13
  10. 2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 12
  11. 2025Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluationcited by 10
  12. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Modelscited by 10
  13. 2025PPTArena: A Benchmark for PowerPoint Editingcited by 9
  14. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 9
  15. 2026RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioningcited by 9
  16. 2026OS-Themis: A Scalable Critic Framework for Generalist GUI Rewardscited by 9
  17. 2024Benchmarking XAI Explanations with Human-Aligned Evaluationscited by 8
  18. 2025LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillationcited by 8
  19. 2026Generative World Renderercited by 8
  20. 2026MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generationcited by 8
  21. 2025PosterForest: Hierarchical Multi-Agent Collaboration for Scientific Poster Generationcited by 7
  22. 2025Human-Agent Collaborative Paper-to-Page Craftingcited by 7
  23. 2026Empowering Reliable Visual-Centric Instruction Following in MLLMscited by 7
  24. 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluatorscited by 6
  25. 2025DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLMcited by 6
  26. 2025KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Languagecited by 5
  27. 2025Understanding Generative AI Capabilities in Everyday Image Editing Taskscited by 5
  28. 2025TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicinecited by 5
  29. 2026VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluationcited by 5
  30. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  31. 2024Can Language Models Evaluate Human Written Text? Case Study on Korean Student Writing for Educationcited by 4
  32. 2024MotIF: Motion Instruction Fine-tuningcited by 4
  33. 2025Instant Preference Alignment for Text-to-Image Diffusion Modelscited by 4
  34. 2025The Mind's Eye: A Multi-Faceted Reward Framework for Guiding Visual Metaphor Generationcited by 4
  35. 2025VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understandingcited by 4
  36. 2025Transferring Textual Preferences to Vision-Language Understanding through Model Mergingcited by 3
  37. 2025REPA: Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilitiescited by 3
  38. 2025Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Chartscited by 3

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
  2. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation
  3. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases
  4. 2025From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens
  5. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
  6. 2026Each Judge Its Own Yardstick: Discovering Per-VLM Taxonomies for Physical Video Evaluation
  7. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
  8. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages