mlsanityData to 7 Oct 2026

Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Models

arXiv 2412.14613 ↗First posted 19 Dec 2024cs.CL cs.AI cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Vision-language models (VLMs) have shown impressive abilities across a range of multi-modal tasks. However, existing metrics for evaluating the quality of text generated by VLMs typically focus on an overall evaluation for a specific task, such as image captioning. While the overall evaluation is essential for any task, the criteria prioritized can differ depending on the task, making it challenging for current metrics to adapt to multi-task scenarios. To address this limitation, we propose HarmonicEval, a reference-free comprehensive evaluation metric that aggregates criterion-wise scores to produce the overall score in a bottom-up manner. Furthermore, to assess the generalizability of automatic evaluation metrics in multi-task scenarios, we construct the Multi-task Multi-criteria Human Evaluation (MMHE) benchmark, which comprises 18,000 expert human judgments across four multi-modal tasks. Our experiments demonstrate that HarmonicEval achieves higher correlations with human judgments than conventional metrics while providing numerical scores for each criterion. Project page: https://stjohn2007.github.io/MMHE_project/

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 55 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

36 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 31

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  11. 2023Large Language Modelscited by 774
  12. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
19 more
  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  2. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  3. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  4. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  5. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  6. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  7. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  8. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  9. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  10. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  11. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  12. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  13. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  14. 2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Modelcited by 44
  15. 2021Transparent Human Evaluation for Image Captioningcited by 40
  16. 2021UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learningcited by 37
  17. 2023InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluationcited by 25
  18. 2024G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4ocited by 23
  19. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extractioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attackscited by 6
  2. 2025Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
  3. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  4. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  5. 2026VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
  6. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models
  7. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  8. 2025Judge Anything: MLLM as a Judge Across Any Modality