mlsanityData to 14 Oct 2026

Swin MAE: Masked Autoencoders for Small Datasets

arXiv 2212.13805 ↗First posted 28 Dec 2022cs.CV cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

The development of deep learning models in medical image analysis is majorly limited by the lack of large-sized and well-annotated datasets. Unsupervised learning does not require labels and is more suitable for solving medical image analysis problems. However, most of the current unsupervised learning methods need to be applied to large datasets. To make unsupervised learning applicable to small datasets, we proposed Swin MAE, which is a masked autoencoder with Swin Transformer as its backbone. Even on a dataset of only a few thousand medical images and without using any pre-trained models, Swin MAE is still able to learn useful semantic features purely from images. It can equal or even slightly outperform the supervised model obtained by Swin Transformer trained on ImageNet in terms of the transfer learning results of downstream tasks. The code is publicly available at https://github.com/Zian-Xu/Swin-MAE.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 28 Dec 2022.

Builds on

31 of 42 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
19 more
  1. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  2. 2018Self-Attention with Relative Position Representationscited by 1,120
  3. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 959
  4. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  5. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  6. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 336
  7. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  8. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  9. 2021SpectralFormer: Rethinking Hyperspectral Image Classification with Transformerscited by 169
  10. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  11. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  12. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  13. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  14. 2022Adversarial Masking for Self-Supervised Learningcited by 78
  15. 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
  16. 2022NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentationcited by 34
  17. 2018A Tour of Unsupervised Deep Learning for Medical Image Analysiscited by 13
  18. 2020Unsupervised domain adaptation based COVID-19 CT infection segmentation networkcited by 5
  19. 2022Parotid Gland MRI Segmentation Based on Swin-Unet and Multimodal Imagescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 5; 2026: 2.

17 papers.

Sort:

By area:

  1. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  2. 2024M4oE: A Foundation Model for Medical Multimodal Image Segmentation with Mixture of Expertscited by 26
  3. 2023TULIP: Transformer for Upsampling of LiDAR Point Cloudscited by 12
  4. 2023Deblurring Masked Autoencoder is Better Recipe for Ultrasound Image Recognitioncited by 8
  5. 2024Generative AI Enables Medical Image Segmentation in Ultra Low-Data Regimescited by 8
  6. 2023EDMAE: An Efficient Decoupled Masked Autoencoder for Standard View Identification in Pediatric Echocardiographycited by 4
  7. 2024VIS-MAE: An Efficient Self-supervised Learning Approach on Medical Image Segmentation and Classificationcited by 4
  8. 2023Pre-Training with Diffusion models for Dental Radiography segmentationcited by 3
  9. 2024Discriminative Hamiltonian Variational Autoencoder for Accurate Tumor Segmentation in Data-Scarce Regimescited by 3
  10. 2025PosDiffAE: Position-aware Diffusion Auto-encoder For High-Resolution Brain Tissue Classification Incorporating Artifact Restorationcited by 3
  11. 2025Endo-CLIP: Progressive Self-Supervised Pre-training on Raw Colonoscopy Recordscited by 2
  12. 2025Global 3D Reconstruction of Clouds & Tropical Cyclonescited by 1
5 more
  1. 2024Unveiling Incomplete Modality Brain Tumor Segmentation: Leveraging Masked Predicted Auto-Encoder and Divergence Learningcited by 0
  2. 2025ModAn-MulSupCon: Modality-and Anatomy-Aware Multi-Label Supervised Contrastive Pretraining for Medical Imagingcited by 0
  3. 2025Weakly Supervised Food Image Segmentation using Vision Transformers and Segment Anything Modelcited by 0
  4. 2026AFFMAE: Scalable Vision Pre-Training for High-Resolution Microscopy Segmentation on Desktop Hardwarecited by 0
  5. 2026Developing a foundation model for high-resolution remote sensing data of the Netherlandscited by 0