# Primus: Enforcing Attention Usage for 3D Medical Image Segmentation

arXiv: https://arxiv.org/abs/2503.01835 · HTML: https://mlsanity.com/paper/2503.01835/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 114 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Builds on (most cited first, 30 of 74)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)

## Built on by (most cited first, 24 of 24)

- [MedDINOv3: How to adapt vision foundation models for medical image segmentation?](https://mlsanity.com/paper/2509.02379/) (2025)
- [Comprehensive language-image pre-training for 3D medical image understanding](https://mlsanity.com/paper/2510.15042/) (2025)
- [Scaling Self-Supervised and Cross-Modal Pretraining for Volumetric CT Transformers](https://mlsanity.com/paper/2511.17209/) (2025)
- [MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentation](https://mlsanity.com/paper/2512.17774/) (2025)
- [Universal CT Representations from Anatomy to Disease Phenotype through Agglomerative Pretraining](https://mlsanity.com/paper/2605.21906/) (2026)
- [Calibrated Self-supervised Vision Transformers Improve Intracranial Arterial Calcification Segmentation from Clinical CT Head Scans](https://mlsanity.com/paper/2507.01744/) (2025)
- [Consistent View Alignment Improves Foundation Models for 3D Medical Image Segmentation](https://mlsanity.com/paper/2509.13846/) (2025)
- [LungEvaty: A Scalable, Open-Source Transformer-based Deep Learning Model for Lung Cancer Risk Prediction in LDCT Screening](https://mlsanity.com/paper/2511.20116/) (2025)
- [Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation](https://mlsanity.com/paper/2608.08713/) (2026)
- [Towards Reliable Pediatric Brain Tumor Segmentation: Task-Specific nnU-Net Enhancements](https://mlsanity.com/paper/2511.00449/) (2025)
- [PlaneCycle: Training-Free 2D-to-3D Lifting of Foundation Models Without Adapters](https://mlsanity.com/paper/2603.04165/) (2026)
- [Effective Feature Learning for 3D Medical Registration via Domain-Specialized DINO Pretraining](https://mlsanity.com/paper/2603.14086/) (2026)
- [SegMaFormer: A Hybrid State-Space and Transformer Model for Efficient Segmentation](https://mlsanity.com/paper/2603.22002/) (2026)
- [FDIF: Formula-Driven supervised Learning with Implicit Functions for 3D Medical Image Segmentation](https://mlsanity.com/paper/2603.23199/) (2026)
- [Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2604.22177/) (2026)
- [ConvNeXt-FD: A Fractal-Based Deep Model for Robust Biomedical Image Segmentation](https://mlsanity.com/paper/2605.22002/) (2026)
- [UniMedSeg: Unified In-Context Learning for Multi-Paradigm 2D/3D Medical Image Segmentation](https://mlsanity.com/paper/2607.12896/) (2026)
- [Beyond scalar losses: calibrating segmentation models via gradient vector field surgery](https://mlsanity.com/paper/2607.14338/) (2026)
- [Whole-Body MRI Classification via Prompt-Based Clinical Conditioning](https://mlsanity.com/paper/2608.30824/) (2026)
- [Evaluating the Generalization of Neuroimaging Foundation Models on African Brain MRI](https://mlsanity.com/paper/2609.23983/) (2026)
- [Brain Metastases Segmentation for BraTS 2026 Task 1: A Multi-Architecture Comparison](https://mlsanity.com/paper/2609.24769/) (2026)
- [nnFoundation: 3D Foundation Models for Radiology](https://mlsanity.com/paper/2609.26924/) (2026)
- [NV-Reason-CT: 3D Visual Language Model for CT Analysis](https://mlsanity.com/paper/2609.27511/) (2026)
- [Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimation](https://mlsanity.com/paper/2610.17573/) (2026)

## Abstract (arXiv)

Transformers have achieved remarkable success across multiple fields, yet their impact on 3D medical image segmentation remains limited with convolutional networks still dominating major benchmarks. In this work, (A) we analyze current Transformer-based segmentation models and identify critical shortcomings, particularly their over-reliance on convolutional blocks. Further, we demonstrate that in some architectures, performance is unaffected by the absence of the Transformer, thereby demonstrating their limited effectiveness. To address these challenges, we move away from hybrid architectures and (B) introduce Transformer-centric segmentation architectures, termed Primus and PrimusV2. Primus leverages high-resolution tokens, combined with advances in positional embeddings and block design, to maximally leverage its Transformer blocks, while PrimusV2 expands on this through an iterative patch embedding. Through these adaptations, Primus surpasses current Transformer-based methods and competes with a default nnU-Net while PrimusV2 exceeds it and is on par with the state-of-the-art CNNs such as ResEnc-L and MedNeXt architectures across nine public datasets. In doing so, we introduce the first competitive Transformer-centric model, making Transformers state-of-the-art in 3D medical image segmentation. The code is available here: https://github.com/MIC-DKFZ/nnUNet/blob/master/documentation/primus.md.