mlsanityData to 7 Oct 2026

GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRI

arXiv 2605.00876 ↗First posted 25 Apr 2026cs.LG cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language models (VLMs) read an image and produce text in a single forward pass, whereas radiologists typically inspect an image several times and consult the literature before writing a report. We introduce GAZE (Grounded Agentic Zero-shot Evaluation), a framework that lets a medical VLM work in this iterative way by calling viewer-level tools (zoom, windowing, contrast, edge detection) and two retrieval tools backed by the U.S. National Library of Medicine (PubMed for medical literature, Open-i for radiological images), with structured outputs validated against a schema and full tool-call traces recorded for auditability. On NOVA, a benchmark of 906 brain MRI cases covering 281 rare neurological conditions, GAZE reaches 58.2 mean average precision (mAP) at intersection-over-union (IoU) 0.3 for lesion localisation and 34.9% Top-1 diagnostic accuracy under a joint protocol that scores captioning, diagnosis, and localisation from the image alone, without task-specific fine-tuning. Before any tool is used, structured prompting and schema-validated outputs already improve over the published Gemini 2.0 Flash baseline (20.2 to 29.4 [email protected]), so framework design is itself an experimental variable. Tool use helps rare pathologies disproportionately: the fraction of cases with IoU > 0.3 rises from 17% to 58% for diagnoses with three or fewer examples versus 25% to 68% for common conditions (≥10 cases), with gains tracking engagement (Gemini 3 Flash: Cohen's d = 0.79, 11.8 tool calls per case; Gemini 2.0 Flash: tools used in 8.2% of cases, no significant benefit). Retrieval ablations additionally reveal a model-dependent trade-off in which gains in diagnosis can coincide with losses in localisation, reinforcing the case for joint evaluation of diagnosis, localisation, and captioning in medical VLMs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 26 references that are in the map. First posted on arXiv 25 Apr 2026.

Builds on

16 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRI20 May
  2. 2024CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting11 Jul

Other references · 14

  1. 2025Qwen2.5-VL Technical Reportcited by 5,418
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  4. 2024Capabilities of Gemini Models in Medicinecited by 231
  5. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  6. 2025MedRAX: Medical Reasoning Agent for Chest X-raycited by 68
  7. 2025ChestX-Reasoner: Advancing Radiology Foundation Models with Reasoning through Step-by-Step Verificationcited by 30
  8. 2025CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answeringcited by 20
  9. 2024VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysiscited by 6
  10. 2025AURA: A Multi-Modal Medical Agent for Understanding, Reasoning & Annotationcited by 5
  11. 2025Agentic Systems in Radiology: Design, Applications, Evaluation, and Challengescited by 5
  12. 2026MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflowscited by 5
2 more
  1. 2025Learning to reason about rare diseases through retrieval-augmented agentscited by 2
  2. 2025Radiologist Copilot: An Agentic Framework Orchestrating Specialized Tools for Reliable Radiology Reportingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Volumetric Radiology AI in the Era of Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders
  2. 2026Confident but Unreliable: A Behavioral Safety Audit of Vision-Language Models on Brain MRI
  3. 2026A World Model of Radiologist Reading for Medical Image Representation Learning
  4. 2025Learning neuroimaging models from health system-scale data
  5. 2026NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding
  6. 2026EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding
  7. 2026Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions
  8. 2025From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentation