# Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding

arXiv: https://arxiv.org/abs/2510.08668 · HTML: https://mlsanity.com/paper/2510.08668/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2025.

## Built on by (most cited first, 30 of 102)

- [CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization](https://mlsanity.com/paper/2511.19661/) (2025)
- [MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution](https://mlsanity.com/paper/2604.26283/) (2026)
- [Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Making](https://mlsanity.com/paper/2602.06570/) (2026)
- [MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning](https://mlsanity.com/paper/2602.03320/) (2026)
- [Real Money, Fake Models: Deceptive Model Claims in Shadow APIs](https://mlsanity.com/paper/2603.01919/) (2026)
- [When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Models](https://mlsanity.com/paper/2603.23085/) (2026)
- [Brain-WM: Brain Glioblastoma World Model](https://mlsanity.com/paper/2603.07562/) (2026)
- [TARSE: Test-Time Adaptation via Retrieval of Skills and Experience for Reasoning Agents](https://mlsanity.com/paper/2603.01241/) (2026)
- [Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine](https://mlsanity.com/paper/2603.06665/) (2026)
- [ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation](https://mlsanity.com/paper/2603.14326/) (2026)
- [ECHO: Efficient Chest X-ray Report Generation with One-step Block Diffusion](https://mlsanity.com/paper/2604.09450/) (2026)
- [Medical thinking with multiple images](https://mlsanity.com/paper/2604.16506/) (2026)
- [MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning](https://mlsanity.com/paper/2605.26567/) (2026)
- [Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory](https://mlsanity.com/paper/2606.09365/) (2026)
- [Medical world models: representing medical states, modelling clinical dynamics and guiding intervention policies](https://mlsanity.com/paper/2606.16721/) (2026)
- [E-MRL: Cross-view Aligned Evidence-driven Multimodal Reinforcement Learning for Reliable 3D Tumor Analysis](https://mlsanity.com/paper/2606.23888/) (2026)
- [Trustworthy and Fair SkinGPT-R1 for Democratizing Dermatological Reasoning across Diverse Ethnicities](https://mlsanity.com/paper/2511.15242/) (2025)
- [Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench](https://mlsanity.com/paper/2601.02737/) (2026)
- [MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis](https://mlsanity.com/paper/2602.22955/) (2026)
- [CodePercept: Code-Grounded Visual STEM Perception for MLLMs](https://mlsanity.com/paper/2603.10757/) (2026)
- [Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs](https://mlsanity.com/paper/2603.20698/) (2026)
- [MedHorizon: Towards Long-context Medical Video Understanding in the Wild](https://mlsanity.com/paper/2605.06537/) (2026)
- [OralAgent: Integrating Reasoning, Tools, and Knowledge for Interactive Dental Image Analysis](https://mlsanity.com/paper/2605.27378/) (2026)
- [Generating Reports or Repeating Templates? Measuring and Mitigating Template Collapse in 3D CT Report Generation](https://mlsanity.com/paper/2605.30984/) (2026)
- [Can Multimodal Large Language Models Understand OCT?](https://mlsanity.com/paper/2607.16609/) (2026)
- [ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding](https://mlsanity.com/paper/2607.24743/) (2026)
- [Radiologist Copilot: An Agentic Framework Orchestrating Specialized Tools for Reliable Radiology Reporting](https://mlsanity.com/paper/2512.02814/) (2025)
- [Learning Transferable Temporal Primitives for Video Reasoning via Synthetic Videos](https://mlsanity.com/paper/2603.17693/) (2026)
- [MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging](https://mlsanity.com/paper/2604.13756/) (2026)
- [How Far Are Video Models from True Multimodal Reasoning?](https://mlsanity.com/paper/2604.19193/) (2026)

## Abstract (arXiv)

Real-world clinical decision-making requires integrating heterogeneous data, including medical text, 2D images, 3D volumes, and videos, while existing AI systems fail to unify all these signals, limiting their utility. In this paper, we introduce Hulu-Med, a transparent, generalist medical Vision-Language Model (VLM) designed to unify language-only, 2D/3D vision-language, and video understanding within a single architecture. Hulu-Med is trained on a curated corpus of 16.7 million samples, comprising exclusively public or synthetic data, spanning 12 major anatomical systems and 14 medical imaging modalities. Hulu-Med employs a medical-aware token-reduction strategy that prunes redundant visual tokens, achieving up to a 55% reduction for 3D and video inputs, improving cross-modal efficiency, and enabling training at 7B-32B parameter scales in approximately 4,000-40,000 GPU hours. Across 30 public in-domain and out-of-domain medical benchmarks-covering text reasoning, visual question answering, report generation, multilingual dialogue, video understanding, and rare disease diagnosis-Hulu-Med surpasses existing open-source models on 27 of 30 benchmarks and outperforms proprietary systems such as GPT-4o on 16 benchmarks. Despite being a VLM, Hulu-Med outperforms GPT-4o and matches GPT-o1 on the text-only HealthBench. For the first time in the community, we provide a fully transparent, reproducible and cost-effective pipeline for holistic medical vision-language understanding by releasing our end-to-end data curation, training procedures, and model parameters. Code and models are available at https://github.com/ZJUI-AI4H/Hulu-Med.