mlsanityData to 14 Oct 2026

Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis

arXiv 2111.14791 ↗First posted 29 Nov 2021cs.CV cs.AI cs.LGcited by 422

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViT)s have shown great performance in self-supervised learning of global and local representations that can be transferred to downstream applications. Inspired by these results, we introduce a novel self-supervised learning framework with tailored proxy tasks for medical image analysis. Specifically, we propose: (i) a new 3D transformer-based model, dubbed Swin UNEt TRansformers (Swin UNETR), with a hierarchical encoder for self-supervised pre-training; (ii) tailored proxy tasks for learning the underlying pattern of human anatomy. We demonstrate successful pre-training of the proposed model on 5,050 publicly available computed tomography (CT) images from various body organs. The effectiveness of our approach is validated by fine-tuning the pre-trained models on the Beyond the Cranial Vault (BTCV) Segmentation Challenge with 13 abdominal organs and segmentation tasks from the Medical Segmentation Decathlon (MSD) dataset. Our model is currently the state-of-the-art (i.e. ranked 1st) on the public test leaderboards of both MSD and BTCV datasets. Code: https://monai.io/research/swin-unetr

Key facts. Cited by 422 papers in the map (data through 14 Oct 2026). Builds on 48 of its 61 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

48 of 61 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  10. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  11. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,887
  12. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
36 more
  1. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  2. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,558
  3. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  4. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  5. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  6. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  7. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  8. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  9. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  10. 2021Video Swin Transformercited by 1,091
  11. 2021Scaling Vision Transformerscited by 999
  12. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 959
  13. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  14. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 639
  15. 2021The Medical Segmentation Decathloncited by 628
  16. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 373
  17. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 368
  18. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  19. 2021Medical Transformer: Gated Axial-Attention for Medical Image Segmentationcited by 320
  20. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 288
  21. 2021CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentationcited by 227
  22. 2016Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challengecited by 201
  23. 20203D Self-Supervised Methods for Medical Imagingcited by 134
  24. 2021DS-TransUNet:Dual Swin Transformer U-Net for Medical Image Segmentationcited by 126
  25. 2021Self-Supervised Learning with Swin Transformerscited by 122
  26. 2021SiT: Self-supervised vIsion Transformercited by 99
  27. 2018SNIPER: Efficient Multi-Scale Trainingcited by 88
  28. 2019Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentationcited by 88
  29. 2021LeViT-UNet: Make Faster Encoders with Transformer for Medical Image Segmentationcited by 85
  30. 2018Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapycited by 77
  31. 2021Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learningcited by 68
  32. 2020SAM: Self-supervised Learning of Pixel-wise Anatomical Embeddings in Radiological Imagescited by 67
  33. 2021DiNTS: Differentiable Neural Network Topology Search for 3D Medical Image Segmentationcited by 48
  34. 2019Scalable Neural Architecture Search for 3D Medical Image Segmentationcited by 21
  35. 2018A multi-scale pyramid of 3D fully convolutional networks for abdominal multi-organ segmentationcited by 17
  36. 2021Self-supervised Image-text Pre-training With Mixed Data In Chest X-rayscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 35; 2023: 103; 2024: 100; 2025: 106; 2026: 76.

422 papers.

Sort:

By area:

  1. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 830
  2. 2022Transformers in Medical Imaging: A Surveycited by 245
  3. 2024nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentationcited by 240
  4. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  5. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  6. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  7. 2021Semi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformercited by 100
  8. 20223D UX-Net: A Large Kernel Volumetric ConvNet Modernizing Hierarchical Transformer for Medical Image Segmentationcited by 94
  9. 2023SegVol: Universal and Interactive Volumetric Medical Image Segmentationcited by 87
  10. 2023MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentationcited by 82
  11. 2023Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-traincited by 75
  12. 2022Masked Image Modeling Advances 3D Medical Image Analysiscited by 72
38 more
  1. 20233DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentationcited by 70
  2. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  3. 2024AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarkingcited by 65
  4. 2024VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imagingcited by 64
  5. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  6. 2023LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matchingcited by 60
  7. 2023Label-Free Liver Tumor Segmentationcited by 56
  8. 2022Medical Diffusion: Denoising Diffusion Probabilistic Models for 3D Medical Image Generationcited by 54
  9. 2023AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weekscited by 52
  10. 2023ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Imagecited by 48
  11. 2023UniSeg: A Prompt-driven Universal Segmentation Model as well as A Strong Representation Learnercited by 47
  12. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentationcited by 47
  13. 2023MultiTalent: A Multi-Dataset Approach to Medical Image Segmentationcited by 45
  14. 2022MSP-Former: Multi-Scale Projection Transformer for Single Image Desnowingcited by 39
  15. 2024BrainSegFounder: Towards 3D Foundation Models for Neuroimage Segmentationcited by 39
  16. 2023SwiFT: Swin 4D fMRI Transformercited by 36
  17. 2024Large-Scale 3D Medical Image Pre-training with Geometric Context Priorscited by 36
  18. 2023AIROGS: Artificial Intelligence for RObust Glaucoma Screening Challengecited by 35
  19. 2023SwinMM: Masked Multi-view with Swin Transformers for 3D Medical Image Segmentationcited by 35
  20. 2024Revisiting MAE pre-training for 3D medical image segmentationcited by 35
  21. 2022TransNorm: Transformer Provides a Strong Spatial Normalization Mechanism for a Deep Segmentation Modelcited by 34
  22. 2023MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generationcited by 34
  23. 2023Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learningcited by 34
  24. 2023Loss Functions in the Era of Semantic Segmentation: A Survey and Outlookcited by 34
  25. 2022Class-Aware Adversarial Transformers for Medical Image Segmentationcited by 33
  26. 2022Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectivescited by 33
  27. 2023Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarkscited by 33
  28. 2023SkinSAM: Empowering Skin Cancer Segmentation with Segment Anything Modelcited by 32
  29. 2024How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Modelcited by 32
  30. 2024Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomographycited by 31
  31. 2024Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?cited by 31
  32. 2025A generalizable 3D framework and model for self-supervised learning in medical imagingcited by 31
  33. 2024Learning General-Purpose Biomedical Volume Representations using Randomized Synthesiscited by 29
  34. 2024How we won BraTS 2023 Adult Glioma challenge? Just faking it! Enhanced Synthetic Data Augmentation and Model Ensemble for brain tumour segmentationcited by 28
  35. 2023Anatomical Invariance Modeling and Semantic Alignment for Self-supervised Learning in 3D Medical Image Analysiscited by 26
  36. 2023MIS-FM: 3D Medical Image Segmentation using Foundation Models Pretrained on a Large-Scale Unannotated Datasetcited by 26
  37. 2024An OpenMind for 3D medical vision self-supervised learningcited by 26
  38. 2023MAPSeg: Unified Unsupervised Domain Adaptation for Heterogeneous Medical Image Segmentation Based on 3D Masked Autoencoding and Pseudo-Labelingcited by 25

50 most cited of 422 All 422 (JSON)