MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Medical Vision-Language Models (Med-VLMs) excel at verbalizing visual content, yet precise visual perception, segmentation, and grounding remain challenging. Existing approaches either verbalize regions as coordinate strings or rely on external modules that decouple perception from understanding, creating representation gaps for region-language alignment. We present MedUP, a Med-VLM that natively unifies perception and understanding within a shared token space. At its core lies UniMedTok, a region tokenizer that encodes masks as discrete tokens in the LLM vocabulary, enabling the model to seamlessly interleave mask tokens with text. We curate UniMed-Train, a 1.84M-instance corpus spanning text-guided segmentation, region-grounded understanding, medical VQA and CoT-based segmentation, and introduce UniMed-Bench for unified evaluation. Extensive experiments show that MedUP outperforms native, agentic, and dual-decoder Med-VLMs across all tasks while remaining competitive with specialist segmentors, demonstrating the strong potential of unified understanding and perception modeling.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 42 references that are in the map. First posted on arXiv 11 Aug 2026.
Builds on
15 of 42 references.
Closest ancestors · 3
Same direction, newest first
- 2025MedSeg-R: Reasoning Segmentation in Medical Images with Multimodal Large Language Models12 Jun
- 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation14 Feb
- 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale27 Jun
Other references · 12
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
- 2025MedGemma Technical Reportcited by 398
- 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
- 2025Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understandingcited by 96
- 2025MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflowcited by 71
- 2025UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretationcited by 10
- 2026IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentationcited by 10
- 2026SAMTok: Representing Any Mask with Two Wordscited by 10
- 2025MediSee: Reasoning-based Pixel-level Perception in Medical Imagescited by 8
- 2025PRS-Med: Position Reasoning Segmentation in Medical Imagingcited by 8
- 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models
- 2026SAM Meets VLM: Parameter-Decoupled Full-Parameter Training for Unified Medical Reasoning and Segmentation
- 2026MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation
- 2026MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation
- 2025OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understanding
- 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
- 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
- 2025Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations