# Multi-encoder nnU-Net outperforms transformer models with self-supervised pretraining

arXiv: https://arxiv.org/abs/2504.03474 · HTML: https://mlsanity.com/paper/2504.03474/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 97 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [Semi-supervised Medical Image Segmentation through Dual-task Consistency](https://mlsanity.com/paper/2009.04448/) (2020)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [On the Challenges and Perspectives of Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2306.05705/) (2023)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [SAM-Med2D](https://mlsanity.com/paper/2308.16184/) (2023)
- [ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset](https://mlsanity.com/paper/2206.06694/) (2022)
- [nnU-Net for Brain Tumor Segmentation](https://mlsanity.com/paper/2011.00848/) (2020)
- [The 2024 Brain Tumor Segmentation (BraTS) Challenge: Glioma Segmentation on Post-treatment MRI](https://mlsanity.com/paper/2405.18368/) (2024)
- [Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation](https://mlsanity.com/paper/1711.01468/) (2017)
- [The Brain Tumor Segmentation (BraTS) Challenge 2023: Glioma Segmentation in Sub-Saharan Africa Patient Population (BraTS-Africa)](https://mlsanity.com/paper/2305.19369/) (2023)
- [BrainSegFounder: Towards 3D Foundation Models for Neuroimage Segmentation](https://mlsanity.com/paper/2406.10395/) (2024)

## Built on by (most cited first, 1 of 1)

- [F3-Net: Foundation Model for Full Abnormality Segmentation of Medical Images with Flexible Input Modality Requirement](https://mlsanity.com/paper/2507.08460/) (2025)

## Abstract (arXiv)

This study addresses the essential task of medical image segmentation, which involves the automatic identification and delineation of anatomical structures and pathological regions in medical images. Accurate segmentation is crucial in radiology, as it aids in the precise localization of abnormalities such as tumors, thereby enabling effective diagnosis, treatment planning, and monitoring of disease progression. Specifically, the size, shape, and location of tumors can significantly influence clinical decision-making and therapeutic strategies, making accurate segmentation a key component of radiological workflows. However, challenges posed by variations in MRI modalities, image artifacts, and the scarcity of labeled data complicate the segmentation task and impact the performance of traditional models. To overcome these limitations, we propose a novel self-supervised learning Multi-encoder nnU-Net architecture designed to process multiple MRI modalities independently through separate encoders. This approach allows the model to capture modality-specific features before fusing them for the final segmentation, thus improving accuracy. Our Multi-encoder nnU-Net demonstrates exceptional performance, achieving a Dice Similarity Coefficient (DSC) of 93.72%, which surpasses that of other models such as vanilla nnU-Net, SegResNet, and Swin UNETR. By leveraging the unique information provided by each modality, the model enhances segmentation tasks, particularly in scenarios with limited annotated data. Evaluations highlight the effectiveness of this architecture in improving tumor segmentation outcomes.