# Auto-Generating Weak Labels for Real & Synthetic Data to Improve Label-Scarce Medical Image Segmentation

arXiv: https://arxiv.org/abs/2404.17033 · HTML: https://mlsanity.com/paper/2404.17033/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1605.01397/) (2016)
- [Diffusion Models for Implicit Image Segmentation Ensembles](https://mlsanity.com/paper/2112.03145/) (2021)
- [UniverSeg: Universal Medical Image Segmentation](https://mlsanity.com/paper/2304.06131/) (2023)
- [Input Augmentation with SAM: Boosting Medical Image Segmentation with Segmentation Foundation Model](https://mlsanity.com/paper/2304.11332/) (2023)
- [KiU-Net: Towards Accurate Segmentation of Biomedical Images using Over-complete Representations](https://mlsanity.com/paper/2006.04878/) (2020)
- [Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasets](https://mlsanity.com/paper/2304.09324/) (2023)
- [Self-Prompting Large Vision Models for Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2308.07624/) (2023)
- [Comprehensive Multimodal Segmentation in Medical Imaging: Combining YOLOv8 with SAM and HQ-SAM Models](https://mlsanity.com/paper/2310.12995/) (2023)
- [Zero-shot performance of the Segment Anything Model (SAM) in 2D medical imaging: A comprehensive evaluation and practical guidelines](https://mlsanity.com/paper/2305.00109/) (2023)
- [Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)](https://mlsanity.com/paper/2311.08077/) (2023)
- [CEmb-SAM: Segment Anything Model with Condition Embedding for Joint Learning from Heterogeneous Datasets](https://mlsanity.com/paper/2308.06957/) (2023)
- [Self-guided Few-shot Semantic Segmentation for Remote Sensing Imagery Based on Large Vision Models](https://mlsanity.com/paper/2311.13200/) (2023)
- [Empirical Evaluation of the Segment Anything Model (SAM) for Brain Tumor Segmentation](https://mlsanity.com/paper/2310.06162/) (2023)

## Built on by (most cited first, 3 of 3)

- [Enabling PSO-Secure Synthetic Data Sharing Using Diversity-Aware Diffusion Models](https://mlsanity.com/paper/2506.17975/) (2025)
- [Understanding Trade offs When Conditioning Synthetic Data](https://mlsanity.com/paper/2507.02217/) (2025)
- [Optimization of MedSAM model based on bounding box adaptive perturbation algorithm](https://mlsanity.com/paper/2503.19700/) (2025)

## Abstract (arXiv)

The high cost of creating pixel-by-pixel gold-standard labels, limited expert availability, and presence of diverse tasks make it challenging to generate segmentation labels to train deep learning models for medical imaging tasks. In this work, we present a new approach to overcome the hurdle of costly medical image labeling by leveraging foundation models like Segment Anything Model (SAM) and its medical alternate MedSAM. Our pipeline has the ability to generate weak labels for any unlabeled medical image and subsequently use it to augment label-scarce datasets. We perform this by leveraging a model trained on a few gold-standard labels and using it to intelligently prompt MedSAM for weak label generation. This automation eliminates the manual prompting step in MedSAM, creating a streamlined process for generating labels for both real and synthetic images, regardless of quantity. We conduct experiments on label-scarce settings for multiple tasks pertaining to modalities ranging from ultrasound, dermatology, and X-rays to demonstrate the usefulness of our pipeline. The code is available at https://github.com/stanfordmlgroup/Auto-Generate-WLs/.