mlsanityData to 14 Oct 2026

ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding

arXiv 2603.19517 ↗First posted 19 Mar 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Everyday photographs taken with ordinary cameras are already widely used in telemedicine and other online health conversations, yet no comprehensive benchmark evaluates whether vision-language models can interpret their medical content. Analyzing these images requires both fine-grained natural image understanding and domain-specific medical reasoning, a combination that challenges both general-purpose and specialized models. We introduce ReXInTheWild, a benchmark of 955 clinician-verified multiple-choice questions spanning seven clinical topics across 484 photographs sourced from the biomedical literature. When evaluated on ReXInTheWild, leading multimodal large language models show substantial performance variation: Gemini-3 achieves 78% accuracy, followed by Claude Opus 4.5 (72%) and GPT-5 (68%), while the medical specialist model MedGemma reaches only 37%. A systematic error analysis also reveals four categories of common errors, ranging from low-level geometric errors to high-level reasoning failures and requiring different mitigation strategies. ReXInTheWild provides a challenging, clinically grounded benchmark at the intersection of natural image understanding and medical reasoning. The dataset is available on HuggingFace.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 23 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

8 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbooks9 May
  2. 2025BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature13 Jan

Other references · 6

  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  3. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  4. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  5. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  6. 2024Gait Patterns as Biomarkers: A Video-Based Approach for Classifying Scoliosiscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  2. 2025TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models
  3. 2026OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models
  4. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM
  5. 2025MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation
  6. 2026M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding
  7. 2026MedObvious: Exposing the Medical Moravec's Paradox in VLMs via Clinical Triage
  8. 2025MedVision: Benchmarking Quantitative Medical Image Analysis