# Unified Medical Image Tokenizer for Autoregressive Synthesis and Understanding

arXiv: https://arxiv.org/abs/2505.19225 · HTML: https://mlsanity.com/paper/2505.19225/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2025.

## Built on by (most cited first, 4 of 4)

- [SciReasoner: Laying the Scientific Reasoning Ground Across Disciplines](https://mlsanity.com/paper/2509.21320/) (2025)
- [Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development](https://mlsanity.com/paper/2603.27460/) (2026)
- [Anatomical Token Uncertainty for Transformer-Guided Active MRI Acquisition](https://mlsanity.com/paper/2603.21806/) (2026)
- [What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization](https://mlsanity.com/paper/2609.24691/) (2026)

## Abstract (arXiv)

Autoregressive modeling has driven major advances in multimodal AI, yet its application to medical imaging remains constrained by the absence of a unified image tokenizer that simultaneously preserves fine-grained anatomical structures and rich clinical semantics across heterogeneous modalities. Existing approaches jointly optimize image reconstruction and textual semantic objectives, relying on large-scale image-caption pairs and are prone to gradient interference. This is ill-suited for the medical domain where paired data are scarce and abundant unpaired images remain unexploited. This work identifies these issues in building unified medical image tokenizers, and introduces a principled two-stage training framework using visual representation as a bridge to address them. The propose visual representation alignment stage enables the utilization of large-scale unpaired medical images to ensure reconstruction fidelity and establish foundational semantics, alleviating the interference and better preparing for the second stage where fine-grained textual semantics are injected using image-text pairs. The resulting tokenizer, MedITok, is trained on over 33 million medical images spanning 9 modalities and 2 million image-text pairs. MedITok achieves state-of-the-art performance on 30+ benchmarks spanning 9 imaging modalities and 4 task families. It further enables autoregressive modeling for diagnostic and generative applications, serving as a scalable component for future multimodal models with unified synthesis and understanding capabilities in the medical domain. Project page: https://github.com/Masaaki-75/meditok