mlsanityData to 14 Oct 2026

Revisiting MAE pre-training for 3D medical image segmentation

arXiv 2410.23132 ↗First posted 30 Oct 2024cs.CV cs.AI cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning (SSL) presents an exciting opportunity to unlock the potential of vast, untapped clinical datasets, for various downstream applications that suffer from the scarcity of labeled data. While SSL has revolutionized fields like natural language processing and computer vision, its adoption in 3D medical image computing has been limited by three key pitfalls: Small pre-training dataset sizes, architectures inadequate for 3D medical image analysis, and insufficient evaluation practices. In this paper, we address these issues by i) leveraging a large-scale dataset of 39k 3D brain MRI volumes and ii) using a Residual Encoder U-Net architecture within the state-of-the-art nnU-Net framework. iii) A robust development framework, incorporating 5 development and 8 testing brain MRI segmentation datasets, allowed performance-driven design decisions to optimize the simple concept of Masked Auto Encoders (MAEs) for 3D CNNs. The resulting model not only surpasses previous SSL methods but also outperforms the strong nnU-Net baseline by an average of approximately 3 Dice points setting a new state-of-the-art. Our code and models are made available here.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 39 of its 63 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

39 of 63 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  7. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  9. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  10. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 830
  11. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  12. 2021The Medical Segmentation Decathloncited by 628
27 more
  1. 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 613
  2. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  3. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 422
  4. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 373
  5. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  6. 2020Contrastive learning of global and local features for medical image segmentation with limited annotationscited by 309
  7. 2024nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentationcited by 240
  8. 2023MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentationcited by 162
  9. 2022ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation datasetcited by 106
  10. 2023STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-trainingcited by 100
  11. 2023Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modelingcited by 94
  12. 2019Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik's Cubecited by 76
  13. 2024VoCo: A Simple-yet-Effective Volume Contrastive Learning Framework for 3D Medical Image Analysiscited by 75
  14. 2022Masked Image Modeling Advances 3D Medical Image Analysiscited by 72
  15. 2022CrossMoDA 2021 challenge: Benchmark of Cross-Modality Domain Adaptation techniques for Vestibular Schwannoma and Cochlea Segmentationcited by 58
  16. 2023MultiTalent: A Multi-Dataset Approach to Medical Image Segmentationcited by 45
  17. 2023Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-trainingcited by 37
  18. 2019Deep Learning Enables Automatic Detection and Segmentation of Brain Metastases on Multi-Sequence MRIcited by 30
  19. 2023MIS-FM: 3D Medical Image Segmentation using Foundation Models Pretrained on a Large-Scale Unannotated Datasetcited by 26
  20. 2023MAPSeg: Unified Unsupervised Domain Adaptation for Heterogeneous Medical Image Segmentation Based on 3D Masked Autoencoding and Pseudo-Labelingcited by 25
  21. 2023Advancing Volumetric Medical Image Segmentation via Global-Local Masked Autoencodercited by 21
  22. 2023vox2vec: A Framework for Self-supervised Contrastive Learning of Voxel-level Representations in Medical Imagescited by 19
  23. 2024AMAES: Augmented Masked Autoencoder Pretraining on Public Brain MRI Data for 3D-Native Segmentationcited by 17
  24. 2023Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-trainingcited by 15
  25. 2024HySparK: Hybrid Sparse Masking for Large Scale Medical Image Pre-Trainingcited by 12
  26. 2024Self Pre-training with Topology- and Spatiality-aware Masked Autoencoders for 3D Medical Image Segmentationcited by 6
  27. 2019COSMOS: Coordination of High-Level Synthesis and Memory Optimization for Hardware Acceleratorscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 21.

35 papers.

Sort:

By area:

  1. 2024An OpenMind for 3D medical vision self-supervised learningcited by 26
  2. 2025MedDINOv3: How to adapt vision foundation models for medical image segmentation?cited by 25
  3. 2025Comprehensive language-image pre-training for 3D medical image understandingcited by 22
  4. 2025Visual Instruction Pretraining for Domain-Specific Foundation Modelscited by 9
  5. 2025General Methods Make Great Domain-specific Foundation Models: A Case-study on Fetal Ultrasoundcited by 7
  6. 2025MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentationcited by 5
  7. 2026Towards Brain MRI Foundation Models for the Clinic: Findings from the FOMO25 Challengecited by 3
  8. 2025Self is the Best Learner: CT-free Ultra-Low-Dose PET Organ Segmentation via Collaborating Denoising and Segmentation Learningcited by 2
  9. 2025The Missing Piece: A Case for Pre-Training in 3D Medical Object Detectioncited by 2
  10. 2025Vehicle-centric Perception via Multimodal Structured Pre-trainingcited by 2
  11. 2025HU-based Foreground Masking for 3D Medical Masked Image Modelingcited by 1
  12. 2026From 100,000+ images to winning the first brain MRI foundation model challenges: Sharing lessons and modelscited by 1
23 more
  1. 2026Finally Outshining the Random Baseline: A Simple and Effective Solution for Active Learning in 3D Biomedical Imagingcited by 1
  2. 2026MeDUET: Disentangled Unified Pretraining for 3D Medical Image Synthesis and Analysiscited by 1
  3. 2026Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentationcited by 1
  4. 2026Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervisioncited by 1
  5. 2026GeoSAE: Geometric Prior-Guided Layer-Wise Sparse Autoencoder Annotation of Brain MRI Foundation Modelscited by 1
  6. 2026Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Studycited by 1
  7. 2024Precision ICU Resource Planning: A Multimodal Model for Brain Surgery Outcomescited by 0
  8. 2025FunduSegmenter: Leveraging the RETFound Foundation Model for Joint Optic Disc and Optic Cup Segmentation in Retinal Fundus Imagescited by 0
  9. 2025Johnson-Lindenstrauss Lemma Guided Network for Efficient 3D Medical Segmentationcited by 0
  10. 2025BertsWin: Resolving Topological Sparsity in 3D Masked Autoencoders via Component-Balanced Structural Optimizationcited by 0
  11. 2026PreSight: Preoperative Outcome Prediction for Parkinson's Disease via Region-Prior Morphometry and Patient-Specific Weightingcited by 0
  12. 2026FDIF: Formula-Driven supervised Learning with Implicit Functions for 3D Medical Image Segmentationcited by 0
  13. 2026CORA: Generalizable coronary artery disease assessment and risk stratification from coronary CT angiography using pathology-centric representation learningcited by 0
  14. 20263D MRI Image Pretraining via Controllable 2D Slice Navigation Taskcited by 0
  15. 2026Benchmarking transferability of SSL pretraining to same and different modality segmentation taskscited by 0
  16. 2026Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelinescited by 0
  17. 2026Report Supervisioncited by 0
  18. 2026A generalizable structural brain MRI foundation model built through dual-priority federated pretrainingcited by 0
  19. 2026KneePreM: Towards 3D Knee MRI Foundation Models via Large-Scale Unlabeled Pretraining and Label-Efficient Fine-Tuningcited by 0
  20. 2026Feature Space Guidance for Breast Cancer Classification in DCE-MRIcited by 0
  21. 2026Anguinus Sculpturae: Compositional Synthesis of Peak-Enhancement Breast DCE-MRI Scanscited by 0
  22. 2026Where to Look Matters: Rethinking Sub-Volume Sampling in 3D Medical Self-supervised Learningcited by 0
  23. 2026LiFUSE: Frequency-Spatial Unified Self-Supervised Pretraining for Liver Disease Analysis on Non-contrast CTcited by 0