mlsanityData to 7 Oct 2026

VividMed: Vision Language Model with Versatile Visual Grounding for Medicine

arXiv 2410.12694 ↗First posted 16 Oct 2024cs.CV cs.CLcited by 28

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Recent advancements in Vision Language Models (VLMs) have demonstrated remarkable promise in generating visually grounded responses. However, their application in the medical domain is hindered by unique challenges. For instance, most VLMs rely on a single method of visual grounding, whereas complex medical tasks demand more versatile approaches. Additionally, while most VLMs process only 2D images, a large portion of medical images are 3D. The lack of medical data further compounds these obstacles. To address these challenges, we present VividMed, a vision language model with versatile visual grounding for medicine. Our model supports generating both semantic segmentation masks and instance-level bounding boxes, and accommodates various imaging modalities, including both 2D and 3D data. We design a three-stage training procedure and an automatic data synthesis pipeline based on open datasets and models. Besides visual grounding tasks, VividMed also excels in other common downstream tasks, including Visual Question Answering (VQA) and report generation. Ablation studies empirically show that the integration of visual grounding ability leads to improved performance on these tasks. Our code is publicly available at https://github.com/function2-llx/MMMM.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 64 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

28 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2020GLU Variants Improve Transformercited by 1,724
  8. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  9. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  10. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  12. 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 597
15 more
  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  2. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  3. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  4. 2022MONAI: An open-source framework for deep learning in healthcarecited by 438
  5. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  6. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  7. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  8. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  9. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  10. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 111
  11. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  12. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 70
  13. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  14. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
  15. 2023Building Universal Foundation Models for Medical Image Analysis with Spatially Adaptive Networkscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 16.

28 papers.

Sort:

By area:

  1. 2024Toward Visual Grounding: A Surveycited by 52
  2. 2025Vision Language Models in Medicinecited by 12
  3. 2025MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precisioncited by 10
  4. 2025CT-GRAPH: Hierarchical Graph Attention Network for Anatomy-Guided CT Report Generationcited by 8
  5. 2026MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMscited by 6
  6. 2026MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentationcited by 4
  7. 2025Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Groundingcited by 3
  8. 2025Trustworthy and Fair SkinGPT-R1 for Democratizing Dermatological Reasoning across Diverse Ethnicitiescited by 3
  9. 2026CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Frameworkcited by 3
  10. 2026Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generationcited by 3
  11. 2025NanoVLMs: How small can we go and still make coherent Vision Language Models?cited by 2
  12. 2026MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUIcited by 2
16 more
  1. 2025CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Trainingcited by 1
  2. 2025Generalised Medical Phrase Groundingcited by 1
  3. 2026AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretationcited by 1
  4. 2026Multi-label Instance-level Generalised Visual Grounding in Agriculturecited by 1
  5. 2025RadDiagSeg-M: A Vision Language Model for Joint Diagnosis and Multi-Target Segmentation in Radiologycited by 0
  6. 2025SpinalSAM-R1: A Vision-Language Multimodal Interactive System for Spine CT Segmentationcited by 0
  7. 2025From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literaturecited by 0
  8. 2026DANCE: Doubly Adaptive Neighborhood Conformal Estimationcited by 0
  9. 2026Tell2Adapt: A Unified Framework for Source Free Unsupervised Domain Adaptation via Vision Foundation Modelcited by 0
  10. 2026Hierarchical Text-Guided Brain Tumor Segmentation via Sub-Region-Aware Promptscited by 0
  11. 2026DGRNet: Disagreement-Guided Refinement for Uncertainty-Aware Brain Tumor Segmentationcited by 0
  12. 2026EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Modelscited by 0
  13. 2026MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Modelscited by 0
  14. 2026Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiologycited by 0
  15. 2026A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scanscited by 0
  16. 2026CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering
  2. 2026How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
  3. 2026VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs
  4. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
  5. 2024Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding
  6. 2026Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
  7. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  8. 2025Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptions