mlsanityData to 7 Oct 2026

VHELM: A Holistic Evaluation of Vision Language Models

arXiv 2410.07112 ↗First posted 9 Oct 2024cs.CV cs.AIcited by 53

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Current benchmarks for assessing vision-language models (VLMs) often focus on their perception or problem-solving capabilities and neglect other critical aspects such as fairness, multilinguality, or toxicity. Furthermore, they differ in their evaluation procedures and the scope of the evaluation, making it difficult to compare models. To address these issues, we extend the HELM framework to VLMs to present the Holistic Evaluation of Vision Language Models (VHELM). VHELM aggregates various datasets to cover one or more of the 9 aspects: visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety. In doing so, we produce a comprehensive, multi-dimensional view of the capabilities of the VLMs across these important factors. In addition, we standardize the standard inference parameters, methods of prompting, and evaluation metrics to enable fair comparisons across models. Our framework is designed to be lightweight and automatic so that evaluation runs are cheap and fast. Our initial run evaluates 22 VLMs on 21 existing datasets to provide a holistic snapshot of the models. We uncover new key findings, such as the fact that efficiency-focused models (e.g., Claude 3 Haiku or Gemini 1.5 Flash) perform significantly worse than their full models (e.g., Claude 3 Opus or Gemini 1.5 Pro) on the bias benchmark but not when evaluated on the other aspects. For transparency, we release the raw model generations and complete results on our website (https://crfm.stanford.edu/helm/vhelm/v2.0.1). VHELM is intended to be a living benchmark, and we hope to continue adding new datasets and models over time.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 49 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

33 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  2. 2024What matters when building vision-language models?3 May
  3. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models15 Mar
  4. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation12 Jan
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  6. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models8 Sep

Other references · 27

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023A Survey on Multimodal Large Language Modelscited by 898
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  12. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
15 more
  1. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  2. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  3. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  4. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  5. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 103
  6. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  7. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  8. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
  9. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Modelscited by 57
  10. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Imagescited by 55
  11. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Imagecited by 50
  12. 2021OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Imagescited by 46
  13. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35
  14. 2023Text-Driven Image Editing via Learnable Regionscited by 31
  15. 2023Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethicscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 39; 2026: 11.

53 papers.

Sort:

By area:

  1. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 40
  2. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 32
  3. 2025Fluid Language Model Benchmarkingcited by 30
  4. 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluationcited by 28
  5. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
  6. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  7. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  8. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  9. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Countingcited by 17
  10. 2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
  11. 2025Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoningcited by 14
  12. 2025Benchmarking Vision-Language Models on Optical Character Recognition in Dynamic Video Environmentscited by 10
38 more
  1. 2025MORALISE: A Structured Benchmark for Moral Alignment in Visual Language Modelscited by 10
  2. 2025Eval Factsheets: A Structured Framework for Documenting AI Evaluationscited by 8
  3. 2025MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understandingcited by 8
  4. 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routerscited by 7
  5. 2025MemLoRA: Distilling Expert Adapters for On-Device Memory Systemscited by 7
  6. 2025Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventionscited by 6
  7. 2025Benchmarking Scientific Understanding and Reasoning for Video Generation using VideoScience-Benchcited by 6
  8. 2025Visual Instruction Bottleneck Tuningcited by 5
  9. 2025Hyperphantasia: A Benchmark for Evaluating the Mental Visualization Capabilities of Multimodal LLMscited by 5
  10. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 5
  11. 2025Medical AI Consensus: A Multi-Agent Framework for Radiology Report Generation and Evaluationcited by 5
  12. 2025Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scalescited by 5
  13. 2024Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Modelscited by 4
  14. 2025Vision-Speech Models: Teaching Speech Models to Converse about Imagescited by 4
  15. 2025Shape and Texture Recognition in Large Vision-Language Modelscited by 3
  16. 2025PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?cited by 3
  17. 2025Re:Verse -- Can Your VLM Read a Manga?cited by 3
  18. 2026SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosiscited by 3
  19. 2026PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Modelscited by 3
  20. 2025Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillationcited by 2
  21. 2025VLM-SlideEval: Evaluating VLMs on Structured Comprehension and Perturbation Sensitivity in PPTcited by 2
  22. 2025Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?cited by 2
  23. 2025Data Matters Most: Auditing Social Bias in Contrastive Vision Language Modelscited by 1
  24. 2025Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospectscited by 1
  25. 2025Robot-Powered Data Flywheels: Deploying Robots in the Wild for Continual Data Collection and Foundation Model Adaptationcited by 1
  26. 2026FaithSCAN: Model-Driven Single-Pass Hallucination Detection for Faithful Visual Question Answeringcited by 1
  27. 2026Visual Persuasion: What Influences Decisions of Vision-Language Models?cited by 1
  28. 2026Compliance-Scored Best-of-N Guardrail Orchestration for Multimodal Document Generation in Payments Dispute Defensecited by 1
  29. 2026Text-to-Image Models Need Less from Text Encoders Than You Thinkcited by 1
  30. 2026Unified Hallucination Fuzzing for Multimodal Large Language Modelscited by 1
  31. 2025Text-to-Image Models and Their Representation of People from Different Nationalities Engaging in Activitiescited by 0
  32. 2025The End Of Universal Lifelong Identifiers: Identity Systems For The AI Eracited by 0
  33. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Featurescited by 0
  34. 2025MEF: A Systematic Evaluation Framework for Text-to-Image Modelscited by 0
  35. 2025Large Language Model Selection with Limited Annotationscited by 0
  36. 2025Reliable and Reproducible Demographic Inference for Fairness in Face Analysiscited by 0
  37. 2025What's in Common? Multimodal Models Hallucinate When Reasoning Across Scenescited by 0
  38. 2026EgoGroups: A Benchmark For Detecting Social Groups of People in the Wildcited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens
  2. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
  3. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  4. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  5. 2025CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models
  6. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?
  7. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
  8. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluations