# Shifting to Machine Supervision: Annotation-Efficient Semi and Self-Supervised Learning for Automatic Medical Image Segmentation and Classification

arXiv: https://arxiv.org/abs/2311.10319 · HTML: https://mlsanity.com/paper/2311.10319/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 17 Nov 2023.

## Builds on (most cited first, 19 of 19)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Uncertainty-aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation](https://mlsanity.com/paper/1907.07034/) (2019)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering](https://mlsanity.com/paper/2103.17070/) (2021)
- [Semi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformer](https://mlsanity.com/paper/2112.04894/) (2021)
- [What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factors](https://mlsanity.com/paper/2203.01825/) (2022)
- [Deep Co-Training for Semi-Supervised Image Segmentation](https://mlsanity.com/paper/1903.11233/) (2019)
- [Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigms](https://mlsanity.com/paper/2007.04234/) (2020)
- [CASS: Cross Architectural Self-Supervision for Medical Image Analysis](https://mlsanity.com/paper/2206.04170/) (2022)
- [A Data-Efficient Deep Learning Framework for Segmentation and Classification of Histopathology Images](https://mlsanity.com/paper/2207.06489/) (2022)
- [Enhancing Medical Image Segmentation: Optimizing Cross-Entropy Weights and Post-Processing with Autoencoders](https://mlsanity.com/paper/2308.10488/) (2023)

## Built on by (most cited first, 5 of 5)

- [DINO-LG: Enhancing Vision Transformers with Label Guidance for Coronary Artery Calcium Detection](https://mlsanity.com/paper/2411.07976/) (2024)
- [Privacy-Aware Continual Self-Supervised Learning on Multi-Window Chest Computed Tomography for Domain-Shift Robustness](https://mlsanity.com/paper/2510.27213/) (2025)
- [SelectiveKD: A semi-supervised framework for cancer detection in DBT through Knowledge Distillation and Pseudo-labeling](https://mlsanity.com/paper/2409.16581/) (2024)
- [Weakly Supervised Pixel-Level Annotation with Visual Interpretability](https://mlsanity.com/paper/2502.17824/) (2025)
- [TRIAGE: Risk-Controlled Pseudo-Label Admission for Annotation-Efficient Semi-Supervised Retinal OCT Classification](https://mlsanity.com/paper/2608.14321/) (2026)

## Abstract (arXiv)

Advancements in clinical treatment are increasingly constrained by the limitations of supervised learning techniques, which depend heavily on large volumes of annotated data. The annotation process is not only costly but also demands substantial time from clinical specialists. Addressing this issue, we introduce the S4MI (Self-Supervision and Semi-Supervision for Medical Imaging) pipeline, a novel approach that leverages advancements in self-supervised and semi-supervised learning. These techniques engage in auxiliary tasks that do not require labeling, thus simplifying the scaling of machine supervision compared to fully-supervised methods. Our study benchmarks these techniques on three distinct medical imaging datasets to evaluate their effectiveness in classification and segmentation tasks. Notably, we observed that self supervised learning significantly surpassed the performance of supervised methods in the classification of all evaluated datasets. Remarkably, the semi-supervised approach demonstrated superior outcomes in segmentation, outperforming fully-supervised methods while using 50% fewer labels across all datasets. In line with our commitment to contributing to the scientific community, we have made the S4MI code openly accessible, allowing for broader application and further development of these methods.