# SwinMM: Masked Multi-view with Swin Transformers for 3D Medical Image Segmentation

arXiv: https://arxiv.org/abs/2307.12591 · HTML: https://mlsanity.com/paper/2307.12591/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 24 Jul 2023.

## Builds on (most cited first, 22 of 22)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation](https://mlsanity.com/paper/1908.10454/) (2019)
- [CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.03024/) (2021)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation](https://mlsanity.com/paper/1803.06815/) (2018)
- [LViT: Language meets Vision Transformer in Medical Image Segmentation](https://mlsanity.com/paper/2206.14718/) (2022)
- [Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation](https://mlsanity.com/paper/2006.16806/) (2020)
- [Multi-Atlas Segmentation of Biomedical Images: A Survey](https://mlsanity.com/paper/1412.3421/) (2014)
- [A Robust Volumetric Transformer for Accurate 3D Tumor Segmentation](https://mlsanity.com/paper/2111.13300/) (2021)
- [Semi-Supervised Multi-Organ Segmentation via Deep Multi-Planar Co-Training](https://mlsanity.com/paper/1804.02586/) (2018)
- [3D Dilated Multi-Fiber Network for Real-time Brain Tumor Segmentation in MRI](https://mlsanity.com/paper/1904.03355/) (2019)
- [MMGL: Multi-Scale Multi-View Global-Local Contrastive learning for Semi-supervised Cardiac Image Segmentation](https://mlsanity.com/paper/2207.01883/) (2022)
- [MVCNet: Multiview Contrastive Network for Unsupervised Representation Learning for 3D CT Lesions](https://mlsanity.com/paper/2108.07662/) (2021)

## Built on by (most cited first, 30 of 35)

- [A Survey of Camouflaged Object Detection and Beyond](https://mlsanity.com/paper/2408.14562/) (2024)
- [Large-Scale 3D Medical Image Pre-training with Geometric Context Priors](https://mlsanity.com/paper/2410.09890/) (2024)
- [Masked Image Modeling: A Survey](https://mlsanity.com/paper/2408.06687/) (2024)
- [Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation](https://mlsanity.com/paper/2404.02845/) (2024)
- [ScribFormer: Transformer Makes CNN Work Better for Scribble-based Medical Image Segmentation](https://mlsanity.com/paper/2402.02029/) (2024)
- [ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embedding](https://mlsanity.com/paper/2307.16226/) (2023)
- [MiM: Mask in Mask Self-Supervised Pre-Training for 3D Medical Image Analysis](https://mlsanity.com/paper/2404.15580/) (2024)
- [Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2412.02621/) (2024)
- [Towards a vision foundation model for comprehensive assessment of Cardiac MRI](https://mlsanity.com/paper/2410.01665/) (2024)
- [Efficient Vision-Language Pre-training by Cluster Masking](https://mlsanity.com/paper/2405.08815/) (2024)
- [Clustering Propagation for Universal Medical Image Segmentation](https://mlsanity.com/paper/2403.16646/) (2024)
- [xLSTM-UNet can be an Effective 2D & 3D Medical Image Segmentation Backbone with Vision-LSTM (ViL) better than its Mamba Counterpart](https://mlsanity.com/paper/2407.01530/) (2024)
- [Vision Foundation Models in Medical Image Analysis: Advances and Challenges](https://mlsanity.com/paper/2502.14584/) (2025)
- [Structure-aware Semantic Discrepancy and Consistency for 3D Medical Image Self-supervised Learning](https://mlsanity.com/paper/2507.02581/) (2025)
- [From Few to More: Scribble-based Medical Image Segmentation via Masked Context Modeling and Continuous Pseudo Labels](https://mlsanity.com/paper/2408.12814/) (2024)
- [Towards a Universal 3D Medical Multi-modality Generalization via Learning Personalized Invariant Representation](https://mlsanity.com/paper/2411.06106/) (2024)
- [Single Image Test-Time Adaptation via Multi-View Co-Training](https://mlsanity.com/paper/2506.23705/) (2025)
- [Foundation Model for Whole-Heart Segmentation: Leveraging Student-Teacher Learning in Multi-Modal Medical Imaging](https://mlsanity.com/paper/2503.19005/) (2025)
- [Boosting Medical Visual Understanding From Multi-Granular Language Learning](https://mlsanity.com/paper/2511.15943/) (2025)
- [An Intra- and Cross-frame Topological Consistency Scheme for Semi-supervised Atherosclerotic Coronary Plaque Segmentation](https://mlsanity.com/paper/2501.07850/) (2025)
- [Unified 3D MRI Representations via Sequence-Invariant Contrastive Learning](https://mlsanity.com/paper/2501.12057/) (2025)
- [MedCL: Learning Consistent Anatomy Distribution for Scribble-supervised Medical Image Segmentation](https://mlsanity.com/paper/2503.22890/) (2025)
- [Opportunistic Osteoporosis Diagnosis via Texture-Preserving Self-Supervision, Mixture of Experts and Multi-Task Integration](https://mlsanity.com/paper/2506.20282/) (2025)
- [HU-based Foreground Masking for 3D Medical Masked Image Modeling](https://mlsanity.com/paper/2509.07534/) (2025)
- [MeDUET: Disentangled Unified Pretraining for 3D Medical Image Synthesis and Analysis](https://mlsanity.com/paper/2602.17901/) (2026)
- [RetiGen: A Framework for Generalized Retinal Diagnosis Using Multi-View Fundus Images](https://mlsanity.com/paper/2403.15647/) (2024)
- [Unveiling Incomplete Modality Brain Tumor Segmentation: Leveraging Masked Predicted Auto-Encoder and Divergence Learning](https://mlsanity.com/paper/2406.08634/) (2024)
- [DiM: f-Divergence Minimization Guided Sharpness-Aware Optimization for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2411.12350/) (2024)
- [ConceptVAE: Self-Supervised Fine-Grained Concept Disentanglement from 2D Echocardiographies](https://mlsanity.com/paper/2502.01335/) (2025)
- [Fake It Right: Injecting Anatomical Logic into Synthetic Supervised Pre-training for Medical Segmentation](https://mlsanity.com/paper/2603.00979/) (2026)

## Abstract (arXiv)

Recent advancements in large-scale Vision Transformers have made significant strides in improving pre-trained models for medical image segmentation. However, these methods face a notable challenge in acquiring a substantial amount of pre-training data, particularly within the medical field. To address this limitation, we present Masked Multi-view with Swin Transformers (SwinMM), a novel multi-view pipeline for enabling accurate and data-efficient self-supervised medical image analysis. Our strategy harnesses the potential of multi-view information by incorporating two principal components. In the pre-training phase, we deploy a masked multi-view encoder devised to concurrently train masked multi-view observations through a range of diverse proxy tasks. These tasks span image reconstruction, rotation, contrastive learning, and a novel task that employs a mutual learning paradigm. This new task capitalizes on the consistency between predictions from various perspectives, enabling the extraction of hidden multi-view information from 3D medical data. In the fine-tuning stage, a cross-view decoder is developed to aggregate the multi-view information through a cross-attention block. Compared with the previous state-of-the-art self-supervised learning method Swin UNETR, SwinMM demonstrates a notable advantage on several medical image segmentation tasks. It allows for a smooth integration of multi-view information, significantly boosting both the accuracy and data-efficiency of the model. Code and models are available at https://github.com/UCSC-VLAA/SwinMM/.