mlsanityData to 14 Oct 2026

ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments

arXiv 2410.06420 ↗First posted 8 Oct 2024cs.CL cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The global shortage of healthcare workers has demanded the development of smart healthcare assistants, which can help monitor and alert healthcare workers when necessary. We examine the healthcare knowledge of existing Large Vision Language Models (LVLMs) via the Visual Question Answering (VQA) task in hospital settings through expert annotated open-ended questions. We introduce the Emergency Room Visual Question Answering (ERVQA) dataset, consisting of <image, question, answer> triplets covering diverse emergency room scenarios, a seminal benchmark for LVLMs. By developing a detailed error taxonomy and analyzing answer trends, we reveal the nuanced nature of the task. We benchmark state-of-the-art open-source and closed LVLMs using traditional and adapted VQA metrics: Entailment Score and CLIPScore Confidence. Analyzing errors across models, we infer trends based on properties like decoder type, model size, and in-context examples. Our findings suggest the ERVQA dataset presents a highly complex task, highlighting the need for specialized, domain-specific solutions.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 43 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

26 of 43 references.

References · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
14 more
  1. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  2. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  3. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  4. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  5. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  6. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
  7. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  8. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  9. 2021Medical Visual Question Answering: A Surveycited by 93
  10. 2020Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretrainingcited by 59
  11. 2023Improving Automatic VQA Evaluation Using Large Language Modelscited by 37
  12. 2020Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questionscited by 19
  13. 2024Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithmcited by 14
  14. 2024Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questionscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoningcited by 14
  2. 2025Better Safe Than Sorry? Overreaction Problem of Vision Language Models in Visual Emergency Recognitioncited by 7
  3. 2025EduVidQA: Generating and Evaluating Long-form Answers to Student Questions based on Lecture Videoscited by 3
  4. 2026Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Actioncited by 2
  5. 2026DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rulescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
  2. 2024SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks
  3. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  4. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
  5. 2025Benchmarking and Mitigating Sycophancy in Medical Vision Language Models
  6. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  7. 2025Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering
  8. 2026Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence