# Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation

arXiv: https://arxiv.org/abs/2605.16393 · HTML: https://mlsanity.com/paper/2605.16393/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 58 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 30 of 35)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentation](https://mlsanity.com/paper/2001.06535/) (2020)
- [nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](https://mlsanity.com/paper/2404.09556/) (2024)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [Spreading vectors for similarity search](https://mlsanity.com/paper/1806.03198/) (2018)
- [BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once](https://mlsanity.com/paper/2405.12971/) (2024)
- [ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset](https://mlsanity.com/paper/2206.06694/) (2022)
- [nnU-Net for Brain Tumor Segmentation](https://mlsanity.com/paper/2011.00848/) (2020)
- [What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factors](https://mlsanity.com/paper/2203.01825/) (2022)
- [Towards Data-Efficient Learning: A Benchmark for COVID-19 CT Lung and Infection Segmentation](https://mlsanity.com/paper/2004.12537/) (2020)
- [Lumbar spine segmentation in MR images: a dataset and a public benchmark](https://mlsanity.com/paper/2306.12217/) (2023)
- [MyoPS: A Benchmark of Myocardial Pathology Segmentation Combining Three-Sequence Cardiac Magnetic Resonance Images](https://mlsanity.com/paper/2201.03186/) (2022)
- [DinoBloom: A Foundation Model for Generalizable Cell Embeddings in Hematology](https://mlsanity.com/paper/2404.05022/) (2024)
- [Boundary-weighted Domain Adaptive Neural Network for Prostate MR Image Segmentation](https://mlsanity.com/paper/1902.08128/) (2019)
- [Pretrained ViTs Yield Versatile Representations For Medical Images](https://mlsanity.com/paper/2303.07034/) (2023)
- [AeroPath: An airway segmentation benchmark dataset with challenging pathology](https://mlsanity.com/paper/2311.01138/) (2023)
- [A Robust Ensemble Algorithm for Ischemic Stroke Lesion Segmentation: Generalizability and Clinical Utility Beyond the ISLES Challenge](https://mlsanity.com/paper/2403.19425/) (2024)
- [Unified Open-World Segmentation with Multi-Modal Prompts](https://mlsanity.com/paper/2510.10524/) (2025)
- [EViT-Unet: U-Net Like Efficient Vision Transformer for Medical Image Segmentation on Mobile and Edge Devices](https://mlsanity.com/paper/2410.15036/) (2024)

## Abstract (arXiv)

Semantic segmentation is essential for analysing anatomical features in biomedical research, yet a performance gap remains for Vision Transformers (ViTs) in the field, particularly for sparse, fine-structured, and low signal-to-noise targets. We attribute this challenge in part to the lightweight pixel decoders commonly used in promptable ViT models, who may lack the local inductive bias needed for high-precision biomedical masks. We bridge this gap by introducing ViTC-UNet, which conditions a UNet on frozen pre-trained ViT representations through learnable tokens and a two-way attention decoder. This combines ViT global visual priors with the local inductive bias and high-resolution decoding capacity of UNets, while avoiding end-to-end ViT fine-tuning even in cross-domain settings. ViTC-UNet outperforms baseline results in semantic segmentation tasks across MRI and CT modalities, demonstrating that structure-conditioned UNet decoding can efficiently adapt large-scale visual priors to high-complexity biomedical segmentation.