mlsanityData to 14 Oct 2026

Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology

arXiv 2606.20477 ↗First posted 18 Jun 2026cs.CV cs.CL cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

We study how to train visually grounded vision-language models (VLMs) for radiology without manual spatial annotations. We introduce RefRad2D, a large-scale bilingual (German/English) dataset of 1.2M CT and MR image-text pairs derived from clinical practice, with task-specific VQA and spatial grounding subsets generated automatically via LLM-based curation and automated segmentation. Trained on this data, our model RadGrounder jointly performs report generation, visual question answering, and spatial grounding via bounding-box detection or segmentation. On external VQA benchmarks (Slake, VQA-RAD), RadGrounder achieves competitive results with specialized medical VLMs. Adding our clinical data to the training mixture improves open-ended VQA over fine-tuning on the downstream datasets alone, showing the transferability of our dataset. Crucially, adding grounding supervision does not degrade language quality, enabling spatially verifiable outputs at no cost to VQA performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 25 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

14 of 25 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024VividMed: Vision Language Model with Versatile Visual Grounding for Medicine16 Oct
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  3. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks26 May

Other references · 11

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Segment Anythingcited by 8,234
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  5. 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 613
  6. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  7. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  8. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  9. 2024TotalSegmentator MRI: Robust Sequence-independent Segmentation of Multiple Anatomic Structures in MRIcited by 59
  10. 2023ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoderscited by 50
  11. 2024Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchycited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI
  2. 2026From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation
  3. 2026LocAnyMed: Vision-Language Grounding for Multimodal Medical Images
  4. 2026U-VLM: Hierarchical Vision Language Modeling for Report Generation
  5. 2026Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
  6. 2026Medical Image Spatial Grounding with Semantic Sampling
  7. 2024Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding
  8. 2023G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training