# MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models

arXiv: https://arxiv.org/abs/2608.10635 · HTML: https://mlsanity.com/paper/2608.10635/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 11 Aug 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 15 of 15)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025)
- [HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale](https://mlsanity.com/paper/2406.19280/) (2024)
- [HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation](https://mlsanity.com/paper/2502.09838/) (2025)
- [Yara Parser: A Fast and Accurate Dependency Parser](https://mlsanity.com/paper/1503.06733/) (2015)
- [Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding](https://mlsanity.com/paper/2510.08668/) (2025)
- [MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow](https://mlsanity.com/paper/2503.18968/) (2025)
- [SAMTok: Representing Any Mask with Two Words](https://mlsanity.com/paper/2601.16093/) (2026)
- [UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretation](https://mlsanity.com/paper/2504.21336/) (2025)
- [MedSeg-R: Reasoning Segmentation in Medical Images with Multimodal Large Language Models](https://mlsanity.com/paper/2506.10465/) (2025)
- [IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation](https://mlsanity.com/paper/2601.03054/) (2026)
- [MediSee: Reasoning-based Pixel-level Perception in Medical Images](https://mlsanity.com/paper/2504.11008/) (2025)
- [PRS-Med: Position Reasoning Segmentation in Medical Imaging](https://mlsanity.com/paper/2505.11872/) (2025)
- [Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning](https://mlsanity.com/paper/2509.19090/) (2025)

## Abstract (arXiv)

Medical Vision-Language Models (Med-VLMs) excel at verbalizing visual content, yet precise visual perception, segmentation, and grounding remain challenging. Existing approaches either verbalize regions as coordinate strings or rely on external modules that decouple perception from understanding, creating representation gaps for region-language alignment. We present MedUP, a Med-VLM that natively unifies perception and understanding within a shared token space. At its core lies UniMedTok, a region tokenizer that encodes masks as discrete tokens in the LLM vocabulary, enabling the model to seamlessly interleave mask tokens with text. We curate UniMed-Train, a 1.84M-instance corpus spanning text-guided segmentation, region-grounded understanding, medical VQA and CoT-based segmentation, and introduce UniMed-Bench for unified evaluation. Extensive experiments show that MedUP outperforms native, agentic, and dual-decoder Med-VLMs across all tasks while remaining competitive with specialist segmentors, demonstrating the strong potential of unified understanding and perception modeling.